Framework Contract SIEA 2018 – Lot 3

Letter of Contract No. 2019/411630

Study on best practices in Third Party

Monitoring

August 2020

Author: Mr. Richard Harrison

The project is funded by The project is implemented by ICE EEIG-led Consortium

the European Union

1
 ICE on behalf of the European Commission

ICE - International Consulting Expertise

150, Chaussée de La Hulpe
B-1170, Brussels, Belgium

Tel: +32.2.792.49.05
Fax: +32.2.792.49.06
www.ice-org.eu

The content of this publication is the sole responsibility of ICE EEIG and can in no way be
taken to reflect the views of the European Union

2
Good practice in conducting Third
Party Monitoring in conflict settings
A research-based handbook for donors and practitioners

3
Table of Contents
Abbreviations .......................................................................................................................................... 6

1 Executive Summary ......................................................................................................................... 7

1.1 Main findings ........................................................................................................................... 7

1.2 Recommendations .................................................................................................................. 8

2 Context .......................................................................................................................................... 10

2.1 Background of this paper ...................................................................................................... 10

2.2 Objectives of this paper......................................................................................................... 10

2.3 Intended audience................................................................................................................. 11

2.4 Methodology ......................................................................................................................... 11

2.5 Limitations ............................................................................................................................. 11

2.6 Notes to the reader ............................................................................................................... 12

3 Considering undertaking TPM ....................................................................................................... 13

3.1 Why do TPM? ........................................................................................................................ 13

3.2 Definition(s) of TPM .............................................................................................................. 15

3.3 When and why to use TPM? ................................................................................................. 16

3.4 General advice on leading TPM ............................................................................................. 19

3.5 Types of TPM ......................................................................................................................... 19

3.6 Models of TPM ...................................................................................................................... 20

3.6.1 The ‘International’ model .............................................................................................. 20

3.6.2 The ‘Local’ alternative model ........................................................................................ 21

4 Tendering and awarding a contract .............................................................................................. 24

4.1 Criteria for selecting a TPM partner ...................................................................................... 24

4.2 Advice on writing and optimising ToRs ................................................................................. 24

4.3 Early Market Engagement ..................................................................................................... 26

4.4 Handling questions ................................................................................................................ 27

5 Inception phase ............................................................................................................................. 28

5.1 The need for Inception phases .............................................................................................. 28

5.2 A blueprint step-by-step approach for managing Inception ................................................. 29

5.3 Communication plans............................................................................................................ 30

4
6 Implementation phase .................................................................................................................. 31

6.1 The typical process ................................................................................................................ 31

6.2 Methodological options – pros and cons of each ................................................................. 33

6.3 How to enhance relationships between all parties ............................................................... 33

6.3.1 Relationship between donors and LAs .......................................................................... 34

6.3.2 Relationship between donors and IPs ........................................................................... 35

6.3.3 Relationship between donors and DCs ......................................................................... 35

6.3.4 Relationship the LA and the IP ...................................................................................... 36

6.3.5 Relationship the IPs and the DC .................................................................................... 36

6.4 Consideration for monitors ................................................................................................... 36

6.5 Implementing Partners and TPM .......................................................................................... 37

6.6 Technology ............................................................................................................................ 38

7 Analysis and reporting ................................................................................................................... 40

7.1 Options for reporting ............................................................................................................ 40

7.2 RAG ratings ............................................................................................................................ 40

7.3 OECD-DAC criteria ................................................................................................................. 41

7.4 Online reporting dashboards ................................................................................................ 41

7.5 Collaboration between donors.............................................................................................. 42

ANNEXES................................................................................................................................................ 44

ANNEX 1 – Bibliography .................................................................................................................... 44

ANNEX 2 – List of contributors .......................................................................................................... 45

ANNEX 3 – Exemplar report structure............................................................................................... 46

5
Abbreviations

AMEL Accountability, Monitoring, Evaluation and Learning

COI Conflict of Interest
CSSF Conflict, Stability and Security Fund
DC Data Collector
DEVCO Directorate-General for International Cooperation and Development
EME Early Market Engagement
EOI Expression of Interest
FGD Focus Group Discussion
HSS Health, Safety and Security
IcSP Instrument contributing to Stability and Peace
IP Implementing Partner
KII Key Informant Interviews
LA Lead Agency
M&E Monitoring and Evaluation
MEL Monitoring, Evaluation and Learning
NFI Non-Food Items
OTI Office Of Transition Initiatives
PoC Point of Contact
PSS Psycho-Social Support
QA Quality Assurance
QC Quality Control
RAG Red, Amber, Green
RFID Radio Frequency Identification
ROM Results Orientated Monitoring
ToR Terms of Reference
TPM Third Party Monitoring
UAV Unmanned Aerial Vehicle
VFM Value For Money

6
1 Executive Summary
1.1 Main findings
On one hand, Third Party Monitoring (TPM) can be described in simple terms as the conducting of
monitoring by a third party – i.e. neither the donor not implementer. Alternatively, it can be seen as
a complex, fast-growing field characterised by a wide variety of forms, all shaped by highly diverse
donor needs, reflecting myriad local contexts. That said, it can still be distilled into two main types –
the monitoring of the views of people, which is seen as more commonplace than the monitoring of
assets. TPM is not only used by donors; it is used by some larger INGOs in high-risk areas where they
themselves lack access.

Two main models of TPM are explained in the report. The ‘conventional model’ sees an international
TPM implementer working as the main interlocutor with the donor. A second ‘alternative model’
sees donors work directly with local organisations. Those who have been involved in various TPM
projects tend to believe that while it is possible to think of a ‘core’ methodology, each TPM
programme’s design tends to be quite unique, because of the diverse requirements (technical, social,
political) placed on the monitoring.

Donor’s motivations for undertaking TPM centre around a lack of access, which is underpinned by a
desire (i) to be accountable (to communities and taxpayers) (ii) to optimise performance and (iii) to
mitigate financial and other risk. There is a school of thought that it should be used only in this ‘last
resort’ sense.

There are clear benefits concerning TPM that go beyond the providing access, in part because TPM
contracts tend to incentivise the private sector to demonstrate considerable levels of innovation,
also through technology. The main benefits are considered to include: the power of high calibre and
independent data, the reduction of project and fiduciary risk, and the ability of TPM to enable
improvement in programming design and refinement. A certain degree of scepticism exists about
TPM, which is driven by the cost, and also by concern around the risk of disempowering of project
managers.

Donors and TPM implementers tend to believe that they enjoy good relations, but face a range of
difficult challenges. The core challenge is that implementing partners (IPs) will naturally be alert to
the accuracy of the TPM findings since (i) they will see it as potentially harming their prospects of
future funding and (ii) TPM implementers are unlikely to have the same level of understanding of the
context and challenges that are faced by the implementing partner in the area visited.

There is a sense that TPM is on the cusp of an acceleration in its evolution through two specific
forms of technology. Respondents noted that aerial imaging data (through satellites and drones) and
‘big data’ could expand the range of data insights that TPM provides while reducing costs, but that
these methods are only now crystallising as users test and validate their utility. Aerial imagery in
particular is expected to provide a lens on visible phenomena on the ground that can be taken as
proxies for behaviours within communities. Dashboards are fairly widely used in addition to
reporting methods, but have seen mixed results in terms of their usefulness.

While most donors acknowledge and appreciate the quality of insights that commercial TPM
providers can bring, they remain highly sensitive to cost, and perceive that TPM implementers can
improve in thinking creatively around this issue.

7
1.2 Recommendations
General advice to donors

Generally, donors should:

 Prepare to play an active role in driving such contracts, ensuring that there is frequent
interaction and a level of trust between all actors.
 Strive for mutual benefit from the data – meaning putting more focus on helping IPs to feel
they receive benefit, encouraging TPM implementers to prioritise the needs of the IPs as
much as the needs of the client.
 Favour up-front-planning including maximal engagement with TPM implementers before
issuing ToRs, including ‘Early market Engagement’ (EME), if in line with the donor’s
procurement policies.
 Choose TPM implementers who have the attitude, chemistry, and local connectivity to
succeed, taking the time to understand as well as possible the internal dynamics and
collective experience of the team under consideration, their interpersonal skills and vision,
and being wary of TPM implementers referred to as ‘body-shops’ who offer a collection of
individual experts, but who may not be well set up to work well together effectively.
 Create exemplar reports (i.e., the deliverable that will be expected of the TPM implementer)
before locking down a Terms of Reference (ToR), as this has been found to be extremely
useful in enabling donors to think through in detail what they do and do not need, so that
this can increase the utility of the ToR.

Donors working in sensitive contexts and/or with sensitive interlocutors

Donors are often obliged to operate in especially fragile and/or sensitive contexts. This raises a
number of challenges, including necessitating operating with a variety of atypical interlocutors,
ranging from UN peacekeeping operations, to niche state security actors (such as intelligence
services), senior and/or niche members of military and police institutions, Civil Society Organisations
(CSOs), politically motivated groups, and armed Non-State Actors (NSAs). Such organisations have
very unique internal dynamics (DCAF, 2015) and may have little or no incentive, motivation or
capacity to be involved in the donor’s TPM/MEL requirements. In such settings, care is taken by
donors to adapt to the circumstances, including by: being alert to the hindrance of the delivery of
aid, pushing for compliance with humanitarian norms, remaining conscious of unique risks (such as
safety of partners), and the heightened risk of misappropriation of funds. In such settings donors are
advised to:

 Be realistic in terms of TPM expectations: Since motivation or capacity to engage with TPM
can be low among some of these actors, donors have to be willing to lower their
expectations. This can mean (i) choosing a narrower range or smaller number of indicators
for the partner to follow – as few as one indicator or measure or (ii) putting a greater focus
on simple, categorical output indicators over complex or qualitative outcome indicators.
 Push for buy-in and access, at a high level: Donors may face resistance in deploying the
standard MEL techniques, or access may be denied to locations or respondents. In such
cases, donors may want to place TPM on the agenda at high-level discussions, perhaps to
ensure that senior leadership commits to a certain basic level of collaboration from the
outset. While such partners may be unaccustomed to TPM, MEL or even evidence gathering,
this does not preclude the possibility that they might be persuaded to play a role. It may be
worth asking them to attend and/or become involved in fieldwork (plausibly to provide
logistical support), and/or to play a role in the dissemination of results. It may also be worth
seeking ‘champions’ within the organisation upon whom success might hinge.

8
  Change the language: Terms such as ‘TPM’ or ‘MEL’ may work well in donor context, but
they may be unappealing to other kinds of actor. Instead, more basic language might be used
such as ‘evidence’, ‘proof’.
 Choose the right representative: Some such actors will listen more closely to personnel on
the donor side whose background mirrors their own. It can be prudent for donors to ensure,
for example, that senior donor military personnel make requests around TPM to senior
military partner personnel.
 Consider aerial technology: Such fragile areas are the environments in which technology
seems to blossom for TPM. Satellite imagery can be used in numerous ways as a proxy for
indicators that might otherwise be captured by monitors. For example one donor is using
satellite imagery to monitor access to a hospital in such a setting, taking this as a proxy
indicator for the flow of its aid to that hospital.

Donors focussed on short-terms interventions

Donors and IPs undertaking shorter-term interventions face a unique set of challenges in terms of
MEL, including that outcomes may only be measurable after the intervention in question has ended.
In these situations, donors should react to this by:

 Increasing the use of qualitative indicators: By its nature qualitative research is better-
placed to pick up on early reaction to an intervention; although the smaller sample sizes used
mean a lack of robustness relative to quantitative methods, the richer, more insightful data,
and ability to use specific sampling, allow for the production of earlier glimpses of real effect.
 Budgeting to return: Donors are encouraged to consider planning to return to certain
locations for one or more subsequent visits in order to undertake further monitoring. Such
visit may seek to (i) evidence if adequate progress has been made (where poor performance
or difficult conditions were previously noted) and/or (ii) identify outcomes and/or impacts
that are attributable to the intervention. Such visits might be conducted by the Third Party
Monitor or, if access has become viable for the donor, by the donor’s own programme
management personnel.

Donors focussed on political outcomes

Donors attempting to realise political benefits face the challenges of trying to measure something
that is of vital importance, yet somehow ethereal and often under-appreciated by the TPM
implementers and implementing partners alike. In these settings donors are encouraged to:

 Be overt about these ambitions, communicate why this is important to all other actors for
example through workshops, and explain that politics equates to power dynamics, and so is
not an ‘ugly’ issue to monitor
 Push the TPM implementers to adapt MEL tools to embrace this issue i.e. build the political
aspect into Theory of Change and logframes.

Third Party Monitoring Implementers

TPM Implementers should bear in mind that:

 Cost remains a key barrier to donors. TPM implementers should dare to push for a clear
budget, but then strive to offer a range of costs that do not only hit the ceiling budget but
offer lower-cost options, and above all make very clear the value and reasoning behind
different options. Ensuring clarity on what is not included in a proposal is also key, so as to
avoid avoidable frustration during the inception phase.

9
  Mind-set – a culture shift is needed by TPM implementers. For these contracts to work, TPM
implementers must consider implementing partners a ‘client’ as important as the donors.
TPM implementers must acknowledge that finally the donor-implementing partners is
usually a more profound and long-term relationship than the donor-TPM implementer, as
there are only a limited number of UN bodies and INGOs.
 Team cohesion: TPM implementers must put more effort on choose teams that are
genuinely cohesive. Various methods may work, such as choosing Team Leaders first, and
encouraging them to develop a vision and build teams from people whom they know will
work well together towards that vision.

2 Context
2.1 Background of this paper
Development, stabilisation and humanitarian workers professionals operate in increasingly conflict-
affected environments. This places limits on donors’ ability to safely visit projects, meaning that
visibility of projects in many vital locations is diminishing. As a result,
the importance of Third Party Monitoring (TPM) is growing (Herbert, Other key
2013), as donors require confidence that their investments are
producing the desired outputs, yielding the desired outcomes and reading on
impact, and adhering to humanitarian norms. The Unit in the EU
Commission’s Service for Foreign Policy Instruments (FPI) responsible TPM (click to view)
for the management of the Instrument contributing to Stability and
Peace (IcSP) commissioned this handbook to add to the body of EU Commission Results
knowledge on this subject, and to help those who need to rely on Orientated Monitoring
Handbook
TPM to understand the reality of the field, and to engage in it with
confidence. DFID Monitoring in Remote
Areas
2.2 Objectives of this paper
USAID TPM document
The principal strategic aim of this paper is to help donors who lead (available upon request)
TPM to do so more efficiently and effectively. TPM involves a range of
Third Party Monitoring in
actors working in highly challenging settings, and this naturally places
Volatile Environments - Do
considerable pressure on those involved and on the processes they the Benefits Outweigh the
use to collaborate. FPI sought to facilitate this process by bringing to Risks?
light the principal issues, challenges and constraints faced by all those
who are involved in TPM. Instruction note for ECHO
staff on Remote Management
The specific aims of this paper are to:
Listening to communities in
 Analyse what constitutes ‘good’ or ‘best’ practice in TPM, as insecure environments
well as identifying practices to avoid
Technologies for monitoring
 Investigate how to identify, procure and work successfully in insecure environments
with a TPM implementer
 Understand how TPM Implementers operate – enabling Between a rock and a hard
donors to work with TPM providers in a manner that is as place: Monitoring aid
convenient as possible to TPM implementers implementation in situations
 Assess what practices can and should be put in place that will of conflict
facilitate a healthy cooperation between TPM implementers
and Implementing Partners (IPs)
 Risk and technical issues connected to this practice of TPM

10
  Explore how to build trust and further the prospects for long term collaboration between
actors
 Highlight the current and potential role that technology plays in TPM.

2.3 Intended audience

This paper is designed principally to be read by donors who lead TPM projects – principally donors
but also on occasions Implementing Partners. The specific target audience is those who are
responsible for leading these projects, and in particular those who may be relatively new to the
responsibility. However, FPI hopes that this report will prove to be of benefit to all involved in the
process of undertaking TPM, by explaining the pressure points in the process, and so facilitating
dialogue to simplify the conducting of good TPM.

2.4 Methodology
There were two components to this research; one primary and one secondary. The first phase was
the desk research – the results of which are summarised in Annex B. This phase sought to understand
what research had already been done on this subject.

The second phase was primary research, which took the form of qualitative Key Informants
Interviews (KIIs). A total of 35 interviews were undertaken, which are categorised in the following
table1:

Respondent type Number interviewed

Donors 16
TPM Implementers 22
Independent experts and academics 4
Total 42

2.5 Limitations
Two principal limitations to this research are noted:

 A number of IPs were invited by the author to take part in this research, however none was
able to do so. This is a limitation since it is logical that the opinions of those whose
organisations are scrutinised by TPM implementers are a key part of the full story of how
best TPM should be undertaken. In an attempt to offset this, TPM implementers who were
interviewed were asked to provide their thoughts on the views of IPs, at least as they see
them. While TPM implementers were willing and able to do this, there is no reason to
believe that the most important views of IPs on this matter are well expressed in this report.
This introduces a risk that this report does not adequately speak to the views of those who
are in the field, and their experience of being monitored in this way. Further research in this
area would therefore be welcomed.
 Owing to COVID 19, which took hold at the same time as fieldwork began, all interviews were
undertaken remotely. Had this not been the case, IcSP may have opted to hold one or more
in-person meetings and/or workshops in Brussels and/or other locations where large
numbers of stakeholders are present.

1
I still hope to interview: Danish MOFA and at least one IP, so this could still go up.

11
2.6 Notes to the reader
 In order to make the report easy to read, the authors have adopted the following key
acronyms for the different actors in the process:
o Implementing Partners ‘IPs’
o Lead Agencies ‘LAs’
o Data Collectors ‘DCs’
 This paper uses the term MEL which may be known by other terms (M&E – ‘Monitoring &
Evaluation’, AMEL ‘Accountability, Monitoring, Evaluation and Learning’ and DMEL ‘Design,
Monitoring, Evaluation and Learning’). These terms are not synonymous but for the purpose
of this report are effectively interchangeable with the term MEL.
 All the quotations in this report were provided by the interviewees.

12
 “TPM is most successful when
you find a way to make the
information it creates
genuinely useful to all parties”
Academic
3 Considering undertaking TPM
TPM is not as complicated as its name might suggest. Simply, it is
MEL done by a third party. However, there are many forms of TPM
and you should situate yourself and your intervention in the broad
spectrum of TPM before starting to consider taking action.
3.1 Why do TPM?
At the most basic level, the principal reason why a donor may want
to undertake TPM is a simple lack of access (Sagmeister et al, “Before writing
2016). In its guidance note on the subject, USAID for example the ToRs, sit
writes:
down and
‘Third-party monitors are contracted by USAID to act as our
eyes and ears when we cannot ourselves access activities’. discuss as a
Interviews explored the reasons why donors have initiated TPM team for hours
project – beyond this simple starting point of access. The main
reason given are summarised and shown in Figure 1. While they
what you really
may be seen as interconnected, some explanation of each is want. The rest
provided:
all flows from
Risk: All development and humanitarian projects carry risk, typically
managed through a risk register. The kinds of risks that are
there”.
addressed by TPM include inadequate design and implementation, Donor
misappropriation, unintended consequences, and conflict
sensitivity. To the Project Manager and those focused on risk management, TPM can become the
principle tool to help mitigate risk (Kelly et al, 2017).

Finance: Invariably, the budgets of programmes considered for TPM are significant, and donors will
be keen to be sure that they are delivering reasonable value for money. With limited or no visibility,
TPM is sometimes undertaken in order to provide a more robust assessment of the financial delivery

13
of the project. This may extend into the undertaking of VFM analysis, a specific field of MEL, which
assesses the ‘4Es’ of economy, efficiency, effectiveness, and equity.

Accountability: Donors strive to be accountable to communities and taxpayers (Chaudri et al, 2017).
IPs invariably have their own MEL systems, but funding and access to qualified local staff for these
can be limited, and the donor cannot easily rely on these IP systems delivering perfectly accurate
insights. If then a donor has neither access nor TPM, and rests only on the data from an IP, this
obligation to communities and taxpayers is placed at some risk. Independently produced and robust
evidence is then helpful in asserting the actual impact of an intervention. It was also emphasised that
it is good practice to link such data to IPs own MEL systems.

Performance: Donors work with IPs to help them to meet the outputs and outcomes committed to in
their proposals. The oversight of these projects is challenging for the IP as well as the donor, and the
contexts in which TPM is undertaken are typically more challenging than a typical project. Donors
and IPs have a shared interest in objectively understanding progress, so that any necessary
improvements can be taken.

Figure 1: Fundamental reasons for undertaking TPM

These four needs are of markedly different importance to different contexts, and, as a result, TPM
interventions take very different shapes (van Beijnum et al, 2018). This makes it challenging for
donors to create ‘libraries’ of templates and tools for the various TPM projects that they manage.
This diversity may also contribute to the difficulty that donors face in collectively understanding and
undertaking TPM.

14
IPs perceive that different donors have different ways of setting about TPM, and that this reflects the
differing context and pressures that each faces. Some donors favour a more scientific and methodical
approach, others attach more weight agility and flexibility. Practitioners see arguments for both, and
favour a case-by-case consideration of the right approach for each TPM programme.

3.2 Definition(s) of TPM

There is no single definition of TPM. Different donors have different definitions of third party
monitoring. The definitions of USAID and DFID, two of the biggest users of TPM are:

USAID: DFID:
Third Party Monitoring (TPM) is the ‘The practice of contracting a third party
systematic and intentional collection of (neither a donor nor implementer) to
performance monitoring and/or collect or verify monitoring data. It is
contextual data by a partner that is not increasingly used to overcome the
USAID or an implementing partner challenges of monitoring in remote or
directly involved in the work. restrictive environments’.

For the purpose of this paper, and leaning on the inputs summarised in Figure 1, we will use the
following definition, which combines aspects of the above and other thoughts provided by the
experts interviewed:

‘TPM is the use of an independent organisation – typically in areas where a funder does
not have access - to collect monitoring data in order to reduce risk and maximise
performance and accountability’.

15
 PUTTING TPM INTO CONTEXT
How to ‘understand’ TPM in the context of …..?
vs. MEL generally? vs. Evaluation?
 MEL is a broader term that describes any  TPM is essentially a process of monitoring,
monitoring, evaluation or learning undertaken not evaluation. The key distinction is that
by any actor. monitoring (and so TPM) is designed to be
 Therefore, we might say that TPM is a subset of on-going.
MEL.  That said, the two do in practice overlap;
 TPM is MEL which is undertaken neither by the this is seen as inevitable and healthy.
donor nor the implementer, and so the  Evaluation is formative and so tends to
independence of the data is a distinguishing draw ‘summative’ conclusions over a longer
factor. That is not to say that TPM data is period of time. Monitoring meanwhile aims
necessarily ‘better’ – IPs will usually know their to be ‘formative’ i.e. to help to form or
environments better, and it is the responsibility shape a programme while it is happening.
of the TPM provider to ensure that it has ample  Evaluation is normally less resource-intense
contextual understanding. and so less costly but only offers feedback
 Learning could be described as reflection which from one moment in time, and – in the case
builds institutional memory and so facilitates of end-line evaluations (the more common
better decision making. Donors increasingly type) too late to adjust a project to
expect IPs to have Learning systems. TPM then enhance performance.
can and should feed into overall Learning.  TPM and evaluation however are
compatible – for large
portfolios/programmes it is considered
normal to do both TPM and evaluation.
“Monitoring should be separate from evaluation because the skill sets are very different; if you bundle
them together, it means that compromises will need to be made” Implementer

vs. ‘ROM’? vs. Project Management?

 Project Management is a function that
 TPM is somewhat broader than Result Oriented
is vital to any intervention, and
Monitoring (‘ROM’)
equally vital to the commissioning of
 ROM is a term that is used by the EU
any TPM
Commission (DEVCO, NEAR), and is specific to
 Where TPM is not undertaken, the
beneficiary-related TPM
oversight provided by Project
 TPM however can include the monitoring of Managers can be seen as striving to
‘people’, ‘assets’ and ‘systems’ – this is the provide a similar function to TPM –
crucial difference between them. i.e. to objectively assess the work of
an IP.

3.3 When and why to use TPM?

When to use

Donors and TPM implementers who were interviewed agreed that the principle drivers to using TPM
is access, i.e. that risks of physical safety and/or travel restrictions (linked to security) often create a
situation where conventional monitoring is not possible. While access is considered the most obvious
and often the fundamental reason to use TPM, it is not the only reason, according to those
interviews. Respondents also put forward the views that TPM should be considered when:

 The location or setting is thought to be especially susceptible to corruption,

misappropriation or theft
 The aid provided includes high-value assets, especially those that could more easily be stolen

16
  The IP has expressed difficulty in undertaking MEL or TPM
 Other niche scenarios – such as threats to the supply chain – e.g. (specific to medicines),
where there is reason to believe that the ‘cold chain’ may be hard to maintain.

Benefits of TPM

In addition to the visibility afforded by TPM where access is otherwise not possible, respondents
pointed to the following core additional benefits of TPM.

 Independence – Information provided by a third party specialising in data collection and

reporting should bring enhanced credibility as the provider is independent and so has no
reason to be unduly favourable to the donor or implementer.
 Technical evidence quality – There is an acknowledgement “I have done
that – if commissioned well and functioning well as a team,
TPM implementers should be capable of providing at least as evaluations as a
high a calibre of data than an IP’s MEL team. While the MEL donor and an
team of an IP may be stretched to appease multiple donors,
and may be stretched for funds, the (typically private sector) implementer
LA should have the resources, means and focus to deliver to
a very high standard.
and I can tell
 Credibility of results – The combination of independence and you, it’s totally
quality should add weight to the robustness of findings. This
should help the donor and IP to feel they can use the results. different -
 Teamwork – Some involved in TPM felt that when TPM is independence is
done well, it can help to establish a culture of agility and
responsiveness between all three key actors – the donor, the everything”
IP, and the LA. If the donor can instil a shared belief that TPM Implementer
benefits all, then it is conceivable to enhance programme
quality and even enhance the team morale.
 Outsourcing efficiency – Another argument for using TPM is to make the work of the donor
more streamlined, alleviating pressure on the donor.
 Local insight – Commissioners of TPM have found that the local teams of enumerators offer
local insight which can add substantial richness to the donor’s understanding of a context.
However, in donors’ experience this needs to be actively harnessed.

Reasons to be cautious about using TPM

Reasons provided for not proceeding with TPM included:

 Risk of disempowerment - it was argued that commissioning TPM can be demotivating

unless carefully handled, and that this would be especially true in situations where the donor
does have access – as this can undermine and damage the role of the Project Manager
 Risk of damaging relationship with IPs – TPM will typically result in issuing a view of IPs’
work. Sometimes, this view can contain more negative feedback than expected. Naturally
then, there needs to be an ample resource to handle this feedback with due sensitivity
 Where the burden on the budget would be too high – commissioners of TPM reported of
occasions when they had no choice but not to undertake TPM because of the unexpectedly
cost
 When there is no reputable provider – examples were given of when a provider simply could
not be found, however it was acknowledged that this could be due to a lack of familiarity
with the market (See Annex B for a list of TPM providers who took part in this research, and
for advice on maintaining a roster)

17
 When the real need can be solved in another way – donors were able to give examples of
when the conventional form of TPM - i.e. sending monitors into the field, can be addressed
in other ways, such as the use of technology. An example was given of using satellites to
show traffic around hospitals to evidence the functionality of a funded facility.

Figure 2: Four varied examples of TPM

EXAMPLES OF TYPES OF TPM FROM AROUND THE WORLD

18
3.4 General advice on leading TPM
Overarching advice to those commissioning and working on TPM is shown below.

10 KEY TIPS FOR LEADING TPM PROJECTS

1. Actively lead: Donor will enjoy the best results if they actively take the lead in the work;
pushing to create a collaborative sense of trust, urgency, and a collective interest in the use
of the data.
2. Overindulge on ToRs: Donors should put all the effort they can into the development of the
ToRs, explain why the project matters. Donors should be open and candid about the real
objectives, what they do and do not want to be told in the report.
3. Last things first: Donors are recommended to draft the report before even drafting the ToRs.
While this may sound counter-intuitive, the intellectual process that this obliges can be
highly illuminating, as it forces team members to think through their expectations.
Describing or even sharing glimpses even of the desired report can make the task of the
bidding agencies far easier.
4. Favour Early Market Engagement: Donors can feel that they sometimes have to issue ToRs
without enough time, information or knowledge of TPM. If and when tendering procedures
allow, donors are advised to take advantage of the option of Early Market Engagement - i.e.
direct interaction with potential bidders as this can and invariably does significantly increase
the quality of the ToR and/or the bids received, and can serve to reduce costs.
5. Demonstrate accountability to communities and monitors: This is where TPM overlaps with
the notion of humanitarian imperatives. In essence, this mean ensuring a constant radar on
the potential for TPM to remain as ‘small’ as possible, to be built on good HSS practices, and
to ask questions that reveal risks to the community.
6. Choose a partner with attitude - as in - the right attitude. Beyond the more obvious
requirements of experience in TPM, methodology, and cost, donors are advised to choose a
TPM partner who has the right attitude towards IPs, towards communication, and can offer
evidence of cohesion within the team and any consortium members (UN, 2015).
7. Don’t let perfect be the victim of good: TPM tends not to operate at the scale of IP
monitoring – as it would be unusual for donors to allocate more funding to ‘verifying
monitoring’ than ‘doing monitoring’. As a result, donors will have to make compromises, and
experience suggests it can be more useful to look more broadly with smaller sample sizes
(Taptue et al, 2017) than to remain fixed on large sample sizes in a fixed set of locations.
8. Visualise: A picture paints a thousand words, and donors can bring reports to life and help
senior stakeholders grasp the key points by evidencing findings through photography,
adhering to standards such as avoiding taking photos of people (or blurring), and removing
meta data.
9. Prioritise vulnerable groups: Donors should encourage TPM implementers to make sure that
they include innovative approaches to ensure ample accessing to women, children and
vulnerable groups.
10. Aim for intimacy: Close and regular, senior communication, on a weekly basis will yield
dividends. Minutes of meetings should be action-orientated and shared quickly.

3.5 Types of TPM

TPM practitioners interviewed agreed that a useful way of looking at the field of TPM is that,
fundamentally, it typically sets out to monitor either people or assets.

People: TPM projects more commonly set out to engage with ‘people’ through qualitative and
quantitative research with:

 Direct beneficiaries – immediate recipients of the aid

19
  Indirect beneficiaries – those who benefit indirectly
 Programme management – the key staff from the implementing partner
 Other stakeholders (such as community leaders, thematic experts) through ‘Key Informant
Interviews’.

TPM practitioners consider that the use of TPM to monitor the opinions of people is significantly
more common than the use of TPM to monitor assets. For this reason the remainder of this report
defaults to discussion of TPM of people, with reference made to assets where applicable.

Assets: In the sorts of fragile and conflict-affected places where access may be impossible and so
TPM more likely to be useful, medium value Non-Food Items (NFIs) or high-value infrastructural
assets may be distributed, and the donor may want to be confident that these remain in the
possession of the intended party/ies. In addition to tangible assets, TPM can set out to look at
intangible assets such as the systems and processes of the IP itself. One relatively common form of
TPM is to assess the strength of IPs’ MEL systems – the logic being that if the donor can be confident
that the IP has the ability to report data well, then there is less need for a high-cost, long term
version of TPM.

Figure 3: Types of TPM

3.6 Models of TPM

3.6.1 The ‘International’ model
Figure 4 shows what practitioners feel has become the ‘standard’, ‘international’ TPM model – i.e.
the one that donors most commonly fund. In this model, the donors sets out to commission a
reputable and experienced lead TPM agency (‘Lead Agencies’ or ‘LAs’) typically in Brussels,
Washington or London, who in turn work with one or more local ‘data collection’ firms (‘DC’s) to visit
the donor’s Implementing Partners (IPs). The number of IPs involved in a typical programme was
described as ranging in number from two to a dozen.

In this set-up, the LA will typically consist of:

 A Team Leader and/or Technical Lead

20
  Experts (often consultants who may be Figure 4: The ‘International’ model
expert in writing, or thematic areas)
 A Project Manager
 A Project Coordinator or Officer
 Cross-cutting experts (such as on Social
Inclusion and/or Gender / vulnerability,
Finance, Conflict Sensitivity).

The LA will direct and be accountable for the work.

It will usually receive the majority of the funds. It
will have prepared (and so absorbed the cost and
risk associated with the proposal-writing), and
identified possible data collection partners, relying
on its networks to understand which providers are
well suited and have the best reputation for
providing quality data. The LA will normally also
provide the intellectual aspects of the deliverables,
and ask the DC to limit its remit to collecting data,
and it will or should put in place processes to
verify the quality of the data collector. Hence –
typically the LA and the DC will intentionally set up
between them something of a governance
structure, in which an ‘invisible line’ is draw
between the two, across which data is handed.
DCs tend to be well-established private sector
companies, often with 5 to 10 or more years of operation. They have dozens or hundreds of
monitors. A small number of regional DCs are thought to exist, and perceptions of their quality vary.
DCs rarely are involved in the writing aspects of the proposal.

“We need to look at a

3.6.2 The ‘Local’ alternative model
working intimately with
In the ‘local’ model, the LA is cut out. The donor works directly with a local [TPM] companies.
DC to collect the data. It’s not just a question of
empowerment, it’s about
The principal advantages of such an approach are (i) the saved cost
shortening the distance
of not having an LA, and (ii) the intimacy between the donor and
those in the field. between the reports and
the field”
The model however does create a risk, which is that DCs may lack Implementer
the skills and capacity to do the work to the standard required. Many
feel this can be remedied by the inclusion of one or more experts.
Careful assessment of DCs, and their financial status, becomes key. “The way the system
Procurement teams may need to assess them in more detail. works now makes it very
hard for Southern TPM
When working with a DC directly, the donor needs to be mindful that
it will implicitly become responsible for checking data quality. The implementers to win
inclusion of an expert then becomes crucial to this model. This contracts. There is an
person is typically very experienced in working for LAs, and can play argument that donors
the role of monitoring their operations, and fulfilling this governance have a responsibility to
role. capacity-build local TPM
providers
Some put forward a view that donors have an obligation to nurture Academic
the development of DCs in this way; that there is such a potential

21
here to contribute to the localization and participatory development agenda.

Figure 5: ‘Local’ Model of TPM

Respondents were able to compare and contrast the relative benefits of both these approaches, and
these perceptions are summarised in the below table:

Figure 6: Pros and Cons of ‘international’ vs. ‘local’ models

Pros Cons
International model  Depth of experience, and so  High cost
credibility  Risk of insights from the field being
 Calibre of work, especially writing lost during report writing
and insightfulness
 Ability to work internationally
Local model  Lower cost, even higher value  Weaker reporting skills
 Local insights (less filtering of the  More risk to donor of being pulled
data) into data processing
 Need to ensure financial viability

TPM Implementers were also asked to self-critique themselves, to reflect on where they feel they
can improve in terms of their service provision to donors.

HOW DO TPM IMPLEMENTERS THINK THEY CAN IMPROVE?

TPM Implementers were asked how they feel they can improve. They said:

 Focussing on monitors and their wellbeing; giving them a clearer stake in the process
 Shifting to see the IP as just as key a ‘client’ as the donor, adopting more of a coaching attitude,
and avoiding coming across as patronising
 Playing an active role in shaping the ToRs
 Working with donors to co-imagine the end deliverable at the beginning of the process
 Helping donors to make the most of Inception phase

TPM Implementers were keen to note that their work is tightly interlaced with the normal work of
MEL and the MEL systems of IPs. They are invariably closely scrutinising logframes and Theories of

22
Change. For this reasons, overleaf, the reader will find a 1-page refresher on Monitoring, Evaluation
and Learning, in order to put issues into context.

Basics of Monitoring, Evaluation and Learning

The field of ‘Monitoring, Evaluation and Learning’ is not without its complexities, but it may still be
paraphrased to a simple question: ‘how is our intervention doing’? In essence, MEL is a set of
perspectives or tools that development and humanitarian actors have developed and refined over a
period of decades to answer this simple question. The field of MEL is constantly evolving; in the 70s
and 80s many spoke of ‘evaluation’ only, then monitoring took shape - underlining the importance of
an ongoing understanding of an intervention’s progress. More recently ‘Learning’ has emerged. Here
is a brief description of what these three inter-linked elements are, how they differ from each other:

 Monitoring – Ongoing measurement of an intervention or programme, typically taking the

shape of data collected by TPM implementers’ own MEL
teams, with some oversight by a donor, in accordance with key
MEL documents, such as the logframe. Conclusions drawn
tend to be based on relatively short interactions. Data may
flows from the project’s own administration, or may be
produced by primary research with beneficiaries, or by field
visits to observe or verify the locations of assets, or the
functioning of a process or system.
 Evaluation – The intermittent assessment of an intervention in
a summative manner. MEL practitioners typically think of
‘baseline’, ‘mid-line’ and ‘endline’ evaluations. These are often carried out by external
consultants, companies or academics, because commissioners see it as essential to ensure
that the more ‘summative’ judgments are truly independent. Often, evaluations will be
based around ‘OECD-DAC criteria (see Section 6).
 Learning – Reflecting from external and internal successes and failures to develop
institutional memory to aid better decision-making in future.

Others feel that accountability is another aspect of the work; i.e. that those undertaking MEL are well
placed to then use that information to ensure that an organisation to remain accountable to its
beneficiaries and taxpayers. Hence, the alternative acronym, ‘MEAL’ that you might be aware of.

Most donors use ‘Theories of Change’ and ‘Logframes’ to underpin their MEL work. The former
present the ‘vision’ of how the intervention will meets its aims, by considering (i) the problems faced
(ii) the outcomes desired, (iii) the outputs and activities that will be needed to bring about this
change. Logframes then detail these elements very specifically, by converting the intended aims into
‘indicators’ which, using ‘SMART’ rules, and by stipulating targets and the ‘means of verification’,
finally provide a solid process which is constantly maintained through each financial year.

 Inputs: The resources to hand to undertake activities (principally: funds and time)
 Activity: Work done to produce each output (some may serve multiple outputs)
 Output: The ‘deliverable’ (a product or service, such as a training session or a document)
 Outcome: The short-to medium-term change intended (also thought of as ‘behaviour
change’) – which the intervention aims to play a significant role in realising.
 Impact: The long-term change intended – acknowledging that other interventions / social
phenomena will play a role in achieving these

23
4 Tendering and awarding a contract
To the extent that tendering procedures allow, TPM implementers
should be brought into the process as early and as closely as possible,
as it helps them to put forward methodologies that will work and be
affordable.
4.1 Criteria for selecting a TPM partner
Donors and TPM implementers were asked to say what they think are the key criteria for choosing a
TPM implementer. Responses are divided into ‘conventional and ‘emergent’.

Figure 7: Criteria for selecting a TPM Partner

Conventional Emergent “The personality of the

 Team cohesion – have the TPM team members is
 Experience of running such team members being put everything. Our
contracts in the location of forward worked together consultant has been
interest before? able to communicate
 Knowledge of the donor,  Assertiveness – do they take positively and earnestly
the country or region, the your ToR and demonstrate with our partners
thematic area are all seen an ability to tell you what without causing
as vital could be improved? problems, and that’s
 Language skills should be  Attitude – in donors’
strong across the team
key”
experience not all Team Donor
 A genuine presence on the Leaders have the right
ground is to be expected characters to instil a sense
 Adequate capacity of confidence in the IPs, and
including for when people this is seen as vital.
are on leave

4.2 Advice on writing and optimising ToRs

The key components of a Terms of Reference are shown below. With ToRs, what
frustrates TPM
 Context implementers?
o A good ToR will include a robust but engaging description of
the context through the eyes of the donors. Most  Unrealistic timelines
importantly, the context should speak to the donor’s  Lack of vision on the
particular view of or link to the context. Most TPM end deliverable
implementers will be highly familiar with the context in a  Inadequate detail
generic sense.  No opinions on
 Rationale methods that might
o The genesis for the project needs to be clear; the problem be used
that the donors wants TPM to solve. There cannot be  No view on how the
enough detail here. It is ideal if the rationale is as open and end product should
candid as possible (i.e. practical or political considerations).
look
 Objectives

24
 o Too often, the objectives are insufficiently thought-through. Implementers are keen to get
under the skin of donors’ real needs, but often sense that there is ‘hidden meaning’ beneath
donors’ stated ‘objectives’. On the one hand the word ‘objectives’ itself has different
underpinnings (‘drivers’, ‘triggers’, ‘influences’, ‘impulses’, ‘rationale’ etc.). On the other, it
can be possible to think in terms of short vs. long-term thinking, and strategic vs. tactical
thinking. TPM implementers want donors to express the fullness of their intention, to go into
as much detail as possible. One simple solution is to split the objectives into ‘strategic’
objectives and ‘technical’ objectives. The box below demonstrates the different kinds of
objective.

Strategic objectives Technical objectives

Summary What senior management will do with What project management will do with
the findings OR the long-term aims OR the findings OR the short to medium-
high-level decision making term aims OR project-level decision
making
Examples  Ensure accountability to  Assess the percentage of
relating to communities beneficiaries who are satisfied
the  Validate claimed IP results  Understand the extent to which the
monitoring  Underpin discussion with intervention is realising benefits in
of people Implementing Partners about terms of relevance, efficiency,
future programming decisions effectiveness, coherence, impact and
 Feed into funding decision-making sustainability
Examples  To reduce the risk of misuse  Percentage and value of lost items
relating to and/or theft  Usability of supplied items
assets  To improve the supply chain

 Non-objectives
o It can be very useful and revealing to TPM implementers to Caution around
know what a donor does not want to know, because they
feel they already know it. For example, it could be that the
Conflicts of
donor feels there is no need to explore for example the Interest (COI)
sustainability of a project, or its conflict sensitivity, if this
information is gathered in other ways. Some of the large
 Criteria international firms act
o Most donors will explain the % of importance that is both as IPs and third party
applied to each aspect of the proposal, of each of the TPM Implementers. This
technical and financial sections. If a little detail can be given can create a Conflict of
on the reasoning behind this allocation of weighting, this Interest, and you should be
will be appreciated by TPM implementers. sure to include in your
selection process a
 Conflict of Interest (COI)
thorough understanding of
o The ToR should instructs the respondent to be clear on
whether the candidate
which projects if there is any, they are already undertaking
organisations undertake
– either as a TPM implementer or as a delivery
implementation in the area
implementer, in the relevant country / region.
in question, or have any
 Issues open to debate
other links that make them
o It is tempting for a donor to write a ToR in such a way as to
inappropriate for the work.
give the impression that everything about the intended
TPM is clear in terms of how it should be done. However, TPM implementers appreciate that
these projects are complex, and it is very hard to be clear on all matters at the point of

25
 writing the ToRs. They would welcome some indication of where there is uncertainty, or
whether there is flexibility to be creative.

4.3 Early Market Engagement

TPM implementers were asked for their views on the process through which engagement happens
with donors in relation to TPM tenders, and what they feel might be done to improve it. The
question was asked in such a way as not to encourage reference to the specific processes and
requirements that each donor has for tendering and bidding for TPM projects.

A key finding from this research is that TPM implementers very much welcome opportunities to
interact directly with donors on the intended work, even to be part of the discussion underpinning
the creation of the ToRs, either in person or remotely. A consensus was found among TPM
implementers for a preference for an invitation to a session at which the project can be discussed
openly – potentially with all interested and/or short-listed bidders present.

Figure 8: Explanation of Early Market Engagement

What it is? A physical or online meeting attended by those interested in bidding for the
work.
When does it Usually, in advance of the formals ToRs being published, if allowed by
happen? tendering procedures.
What is the Typically donors will start by presenting draft ToRs. These may or may not be
format? shared in advance. If an Expression of Interest (EoI) was issued, it may be
sensible to show the basic ToRs likely shared at that point.
Who is involved? Any TPM implementer – depending on how large a field the donors wants to
invite. Donors may choose to invite only those who have passed a ‘first round’
of consideration, perhaps through the submission of EoI that looks at basic
criteria such as track record, financial health, presence on the ground.
Why it is useful? It provides a chance for donors and TPM implementers to genuinely discuss
the issues at hand. It can greatly increase the chances of donors feeling that
the formal, final ToRs are well constructed. It gives TPM implementers a
chance to ask questions that can help them to reduce their costs.
Any risks? TPM Implementers feel these meetings can be less useful or less appealing to
them when the format tries to oblige or coerce the attendees to share their
ideas, and so lose their competitive edge. Large LAs have often established
leadership positions by developing intellectual property around TPM which
they want to protect.

Maintaining a roster of TPM consultants and firms?

An option that has been considered and used by some is to maintain rosters of firms of TPM
implementers, and/or rosters of consultants who could play a role in a ‘local’ model. Interviewees
felt that for this to work it has to be actively managed i.e. assigned specific responsibility to an
individual who coordinates with commissioners to (i) search on an ongoing basis for providers –
including through asking contacts (ii) collating financial and technical scores of those who submit bids
and (iii) making available the database to everyone interested in commissioning such work. The
possibility of creating and maintaining a roster of consultants should, however, take into
consideration EU regulations on data protection; TPM implementers should ensure that they comply
with EU rules on this subject.

26
4.4 Handling questions
Both donors and TPM implementers understand that – no matter how thorough the ToRs are –
bidders will want to ask questions during the bidding process. TPM Implementers perceive that
donors commissioning TPM generally do invite the asking of questions, which is welcomed. Without
being asked to draw comment on the specific processes and requirements that individuals donors
have in place for tendering and bidding TPM projects, TPM implementers were asked for ideas for
how the process for asking and answering of questions could be improved, and have these
suggestions:

 A commitment to a transparent process in which responses to all questions are shown to all
bidders
 Donors allowing at least a 2-week window in which questions are asked and answered
 Donors aiming to respond to each question within 3 working days
 Consideration of a web-based multi-user interface to (i) allow for clear communication of the
answers and reduce reliance on emails (which can cause confusion around the most recent
response to each question) and (ii) reduce the risk of donors receiving the same question
from multiple bidders
 Allowing follow-up questions (within the pre-agreed timeframe).

27
5 Inception phase
Nearly all TPM projects have to adapt considerably from their initial
design – as these are highly complex environments. Piloting the
methodology is vital, and if the donor team is going to invest heavily
at any point in the process, it should be here.
5.1 The need for Inception phases
Most donors and TPM implementers are used to the idea of pilot or inception phases, and these are
seen by TPM implementers and donors as vital for TPM projects also. Respondents felt that inception
phases are vital because:

 The basic context in which the TPM operates is so volatile that even the best-informed
bidder will not be able to put forward a
planned methodology in which having full Good documentation
confidence;
 It is plausible that bidders did not genuinely For a TPM implementer to be ready to
have enough capacity to think through go into the field, they need to have in
every aspect of their proposal, and they will place a set of documents that will
need more time to test their own assertions ensure that everyone – and the
in the proposal; and monitors especially – has the same
 TPM implementers admit that they may thorough understanding of their roles
sometimes put innovative ideas into their and responsibilities. It can be tempting
proposals that are included to attract for an implementer to let these
donors’ attention and help them win – these documents evolve during the inception
cannot always be fully thought-through or phase. Experience shows it is better to
fully-costed out, and this needs to be push for TPM partners to prepare a full
acknowledged and explored together with ‘V1’ set of these documents before
the donor. fieldwork, and then to formally review
them after a reasonable number of
TPM implementers understand that donors need to visits is undertaken.
ensure that the procurement process is fair, which
Also, since these documents can be so
means in effect ‘keeping bidders at arm’s length’
time-consuming to make, it is a good
during the process. A side-effect of this approach
idea to have them in place during the
however is that – as a result - there tends to be no
relative calm before implementation.
substantive direct contact between donor and
bidder at any point in the process (unless there is an  Methodology manual
EME). In turn, this can result in dissonance or even  Monitor manuals
misunderstanding between the ‘winning’ bidder and  Health, Safety & Security manual
donor, especially if the donor lacks capacity in the  Monitor training plan
weeks after the bid is won.

TPM implementers then hope to have the opportunity for a highly collaborative approach to
inception, during which they and the donor acknowledge that they have (typically) arrived at this
point with limited contact, and need to spend time together and have an open, candid discussions
about the proposal.

In the TPM field, unlike some others industries (such as advertising) where large contracts are
tendered for, bidding companies are not paid for their time in preparing the proposal. As a result,
they have no choice but to absorb the cost of bidding. They do not have limitless capacity, and

28
cannot be expected to put forward a ‘perfect’ methodology. Therefore, at the point at which a
tender is awarded, it is obvious that the winning company itself may well be aware that it might have
gone further, been clearer, or costed more precisely, some aspects of its bid.

5.2 A blueprint step-by-step approach for managing Inception

Respondents were asked for their views on how to navigate the inception phase, and gave the
following thoughts.

Figure 9: Blueprint step-by-step approach for managing Inception

Stage Content

Provisional One of the sensitivities of this part of the process is a possible tension between (i) the
1 award ‘winning’ bidder being announced and (ii) a likely need to make concrete changes to the
methodology that may result in additional costs. Specifically, this situation can result in
the LA being asked to provide further quotations for costs what has by then become a
non-competitive setting. For this reason, it may be the case that donors will want to
retain some control in the process by awarding only a ‘provisional winner’ and
announcing a final decision only once a full dialogue has been undertaken. This allows
both parties an opportunity to make sure that they are a good fit, and to ‘iron out
creases’ in the putative methodology. Such a phase is also useful as it allows for appeal
against the decision (if the donor operates such a system). In any case, at this point, IPs
should be made aware of the provisional award, and ideally they would be informed
ahead of any public announcement, as a courtesy.
Cost As described above, it may be in the interest of the donor, having nominated the
2 clarification ‘provisional’ winner, to enter into detailed discussions about the proposal and the costs,
> formal inviting reflections from the winning bidder about anything that they were unsure
awarding. about. The donor may want to bring in MEL experts to scrutinise bids, possibly to help
the donor team to ask questions to make sure that every aspect of the financial side of
the project has been well thought-through.
Methodology Once the financials are complete, and the donor has confidence that there are no
3 development surprises in terms of finances, and the formal award has been made, the donor and LA
should come together and discuss in detail the methodology. If viable, follow-up
technical meetings may be agreed to finalise these. A suite of tools should be
developed; this may take some weeks to prepare. An initial communications plan may
also be drafted during this time. It is also sensible to consider a detailed documentation
of the steps of fieldwork including precisely who is expected to do what, and when.
IP Once the ‘pilot-methodology’ is readied, it should be shared with the IPs. This is a crucial
4 Engagement moment in the chronology of the project – it is the first time that the donor will present
to the IPs the real nature of the project. IPs may have many useful ideas about the
proposed methodology – they may well want to share these. This engagement should
be done in such a way as to send a signal to the IPs that the donor treats both parties
equally and requires a proactive and cohesive approach. This stage may include one-to-
one meetings, a workshop or a combination of both, whether in person or online.
Fieldwork With the methodology now endorsed by the IPs, and the expectations on them clear, a
5 begins first visit to the field should be undertaken. This would ideally be undertaken with more
experienced monitors (who are briefed to feed back in a fulsome way) and with plenty
of time to reflect afterwards on potential improvements. Once this is done, a second
and if needed third wave of piloting is recommended, testing also a frequency of
fieldwork that it likely to reflect normal speed of operation.
Finalising of Once the team believes that ample fieldwork has been undertaken, all tools should be
6 methodology revised and formalised with donor approval. A formal ‘methodology’ handbook should
be approved and signed off by the donor. This should include clarity on all important
process such as scoring and definitions (see Implementation section).

29
Through all of the above, TPM practitioners emphasise that this Inception phase provides a crucial
first opportunity for coalescing of the ‘full team’ that is the donor, the LA and the IP. In this sense it is
vital that the donor sets the right tone, and makes time available, including by senior colleagues.

5.3 Communication plans

The inception phases is a good point at which to consider rolling out a Communication Plan; i.e. a
simple – possibly 1 or 2-page agreement around who will contact, and when. The key rationale for a
Communications Plan is that:

 IPs are busy and it may help them to be presented with a very clear plan, which aims to
minimise the logistical burden on them
 TPM reports are often sensitive in nature
 It can happen that the DC finds an urgent problem (See Section 5 for more on ‘Red Flags’)
and clear communications is needed for such situations
 In the more fragile settings, where monitor security is a concern, clear communications plans
are essential to make it clear who will play what role in an emergency

That said, not all projects are thought to require such a plan, and it may be a good strategy to wait
and see if this is even needed. Experience shows however that TPM of larger programmes, of more
sensitive work, or ones where more challenging information may be expected to be shared
(especially between LA and IP) may benefit particularly.

A good Communications Plan should:

 Be as concise as is reasonable
 Be scenario-specific
 Account for all scenarios that relate to safety
 Involve senior decision makers.

30
 “TPM will only work if the
donor sets the right tone –
explains with conviction why
this is important, and what
everyone needs to do and not
do to make it work”
Implementer
6 Implementation phase
During implementation, frequency of communication between all
parties is crucial; try to emphasise the need for open, frequent
communication around a clear fieldwork plan, and to instil a common
appreciation of the main aim – performance improvement to the
benefit of all
6.1 The typical process
The implementation of TPM typically involves months or years of interaction between the donor, LA,
DC and IPs. As described in Section 2, there are many types of TPM. In this section, we will look at
how implementation may look for two of the principal forms of TPM; that undertaken with
beneficiaries, and of MEL systems.

Overleaf we see a description of a typical flow of each kind of project.

“The most successful TPM I ever worked on brought everyone

together frequently to discuss improvements and learning
often and authentically”
Implementer

31
Figure 10: Typical TPM implementation process; with beneficiaries, and of MEL systems

32
6.2 Methodological options – pros and cons of each
Donors and TPM implementers both understand that primary (qualitative and quantitative) research
is often a necessary aspect of TPM. The table below expands on perceptions of each and their utility
in the field of TPM.

Figure 11: Generic explanation of methodological options, pros and cons

Why and how to use Challenges in using

Qualitative:  These have become the most  They can take a long time to administer;
Depth interviews common approach to obtaining the even if the intention is to speak for just
with community views of beneficiaries and community 20 to 30 minutes, individuals may want to
members members during TPM, because they speak for longer, and so completing a
provide rich insights, allowing daily quota can be difficult.
respondents to tell the full,  In highly conflict-affected areas, open
sometimes complex story of their conversations can be especially likely to
engagement with the intervention. open up feelings that may even require
Psycho-Social Support (PSS).
Qualitative:  When undertaking TPM regarding the  The right balance of individuals is key; it is
Depth interviews progress on an intervention, it is unhelpful to speak to only senior or only
with project sensible and necessary to speak with junior colleagues.
team members those implementing the work. No  Where the intervention relies on highly
TPM can claim to be balanced without technical staff (such as a surgeon in a
this. field hospital) – interviewing this person
 Project team members can be is key.
interviewed remotely, and it can make  Tools have to be carefully crafted, and
sense before a deployment with one monitors trained, to avoid overly-
or two key individuals. defensive responses.
Qualitative: Key  Key Informant Interviews with other  These can be difficult to arrange in
Informant actors (such as community leaders, certain settings, and on occasion
Interviews other INGOs) are crucial to politically sensitive.
understanding the broader context
and so highly useful for enabling the
donor to be confident about the
relevance and impact of an
intervention.
Qualitative:  FGDs ‘strength in numbers’ may make  In some communities men and women
Focus Group it easier to talk frankly. cannot come together.
Discussions  Quality of discussion can be higher as  Logistical challenge of scheduling multiple
a result of interaction of numerous people at one time
people.
Quantitative:  Robust, categorical evidence of  Cost and time of large sample sizes.
Surveys progress of an intervention.

6.3 How to enhance relationships between all parties

Figure 12, below, summarise the key ways in which respondents suggested that relationships
between each party could be enhanced. The diagram is then explained below.

33
 Figure 12: Ways to strengthen relationships between actors

6.3.1 Relationship between donors and LAs “One IP said to

Senior involvement is key to this relationship working. TPM me – can you
implementers feel that if the donor ensures that senior
management are occasionally involved, they have the ability to
please tell the
ensure that all stakeholders understand that the work is important. donor how hard
As a result, the perception is that actions are more commonly
taken. this data
In addition to this, the use of a Communication Plan is essential to collection is? It’s
guarantee that there are no significant miscommunications. The more powerful
relationships in this situation are sensitive – the donor will be
mindful of assuring that the relationship with the IP – which will coming from
likely extend to other programmes and regions – is not unduly
affected by miscommunication. Therefore, the donor must stress
you”
to the LA and/or DC that the Communication Plan, if one is in place, Implementer
must be understood by all and followed to the letter, to avoid any
damaging of relationships.

34
Financial matters often run at the heart of this relationship. The LA will have won the project either
at a margin that it is pleased with or, perhaps in order to gain prominence in the field, at a lower
margin, and so may be sensitive to any new work or requests that are not funded. It is essential then
that the donors and LA talk openly about finances, and although the profitability of the LA is no
concern of the donor, it will be beneficial if both parties can achieve an open and frank discussion on
the underlying state of finances. Ideally, LAs are invited to explain what their cost drivers are, and for
a conversation to be ongoing around how both parties are seeing the financial side. The degree to
which budgets are flexible – i.e. the donor’s tolerance for the LA to make unilateral decisions, should
also be made clear.

6.3.2 Relationship between donors and IPs

The relationship between the donor and the IP is arguably the most important and can be highly
sensitive (Rivas et al, 2015). Crucial to this, practitioners feel, is allowing the IP to have first sight of,
and a fair amount of time to review, a first draft of the report. It can often be the case that the DC
will not have had all the time it would like to understand the project it has visited; these projects
would ideally involve a day or more of briefing from the IP but this is not realistic for most IPs.
Therefore, reports can be submitted with flaws due to understandable resource issues, and the IP
should be given the chance to counter any such flaws, before the reports go to the donor and create
misunderstanding. TPM implementers hope that donors can help to work with the IPs to create an
intimate, efficient environment that is perhaps more dynamic than a normal MEL environment,
creating a shared interest in guiding and shaping a programme through the use of independent,
insightful, recent data. They hope that donors can encourage IPs not to look at TPM team as ‘the
policeman’, but rather as a partner in building the legitimacy of the programme.

6.3.3 Relationship between donors and DCs “The best

In the ‘conventional’, ‘international’ approach, where a project has an learning I have
LA and a DC, there is less clear reason why the donor and the DC
would interact as the LA typically plays the role of the intermediary. It
seen is done
is advisable however that the donor insists on a clear line of slight to over coffee”
the DC, perhaps on the basis of joining monthly or quarterly
meetings, to ensure that the DC feels that its views, and any Implementer
problems, are fully known to the donor. Given that the LA and DC, as
explained in Section 2, usually default – in a governance sense – to
the roles of ‘data provider’ and ‘data checker’, tensions can arise
RED FLAGS
between the two. Finances can also be an issue – the LA is likely to Sometimes TPM
have a number of expensive international consultants working in safe implementers may find
locations, while the DC is managing a range of relatively low-paid something seriously wrong,
monitors and managers. It is important that the donor be cognisant such as an observation of a
of the potential tension, and at least keeps the line of communication clear risk to children. For
open with the DC in order to mitigate this risk. Where the donor is this reason it is sensible to
working directly with a DC, there is a greater likelihood that the data consider putting in place a
received by the donor will be in a more basic form, or less polished. If process through which such
this is the case, the donor will need to ensure that it dedicates ample findings are ‘red-flagged’ –
project manager or coordinator-level resource to work more closely i.e. the DC is instructed to
with the DC to explore the data. send an immediate
notification either to the IP
or donor.

35
6.3.4 Relationship the LA and the IP
There is a clear potential for tension between the LA and the IP, as the latter is being evaluated by
the former. The donor can assist this relationship by:

 Communicating clearly and as soon as possible about the purpose of the TPM programme,
and how the information will be used, ideally as soon as the programme is decided upon.
This may involve the use of a workshop MEET THE MONITOR - TYPICAL DAY OF A
with other IPs. THIRD-PARTY MONITOR INSIDE SYRIA
 Ensuring that any scoring (See Section 6)
e.g. any RAG-ratings are well defined and 7am – I woke up and checked Twitter and
justified, and that the tools that the Signal to see what’s happening around my
programme utilises are well thought- area, to see if it’s safe to head out. Yes,
through. there have been a few strikes over night,
 Setting up processes and meetings during but nothing on the route I plan to take.
the course of the TPM programme in which
the donor actively plays a role in 8am – I called my supervisor, as agreed. We
emphasising the importance of TPM, and compared notes about the strikes, we’d
the creation of a culture of feedback and heard slightly different things, and we
learning, to improve performance. agreed to message a few more people
before we decide whether it’s safe for me
6.3.5 Relationship the IPs and the DC to head out and do the visit or not.

Relationships between the IP and the DC centre 9am – I should have left already but needed
around the visit itself. Often the IP’s staff at the that time to make sure things are safe – I
location to be visited will be very busy, and may feel they are and so does my line manager
struggle to guarantee being free during the visit and male colleague.
itself. The donor can help by stressing the
importance of the IP’s management asking the local 9.30am – Say goodbye to family and head
team to be free, and urging them to make sure that to the destination. I remind myself how to
to the extent possible, all key stakeholders are delete all my data if I get stopped at a
present, and that key activities can be observed. checkpoint.

11am – Arrive 45 minutes late at the

6.4 Consideration for monitors location, as I missed the bus I was
Monitors are the most essential part of the TPM expecting to get. Had to take a taxi. Need
process. They are the ones that undertake the core to know if I can reclaim that cost, The IP
work – the data collection - and in war-affected PoC is upset as I’m late and she doesn’t
settings, put themselves in harm’s way to do so. really have time to brief me. We struggle
Therefore, their wellbeing is a clear ‘Do No Harm through.
that’ priority for everyone involved in the TPM. 12 noon to 3pm - We did the fieldwork,
Monitors are typically well-educated and expert in everything went basically fine, but we
administering qualitative or quantitative achieved fewer interviews than we’d
interviews. They are – these days – used to working hoped.
with devices, but may sometimes work with audio 3pm - Return home on time, early because
recording or paper if there is need. travelling at night is not wise. Confirm my
Given the risks that monitors take, the culture of safe arrival back home. Start analysing the
the team needs to be geared around their safety. data.
Monitors must have the right to refuse to deploy,
and must not be coerced into doing so. In conflict-affected areas especially, particular care must be
taken to produce a full set of Health, Safety and Security (HSS) documents that make clear to the

36
monitor every aspect of the process, and what they must do in certain settings, such as if they are
asked who the donor is, or if they are questioned at a checkpoint. Technology can be used to help
monitors – for example there are now software that can hide the apps if monitors are stopped by
the authorities or threatened.

No matter how experienced they are, monitors will crave high quality briefings and trainings. These
are essential and donors may want to ensure that these are well executed. Trainings typically take
two forms – initial ‘generalised’ training on the project, and safety protocols and ‘visit specific’
training on the IP and their work, the tools, and what to expect on the day. Donors can help by
reviewing the materials, or even joining some trainings to make sure they are to standard.

6.5 Implementing Partners and TPM

Implementing partners are seen as being open to TPM, and usually making earnest effort to make
best use of the data. They understandably object strongly to anything they consider to be
inaccurate, citing examples where TPM implementers have failed to take into consideration either
the full context of the work either in terms of the local reality, the thematic subject matter, or the
modality of the implementer. There is an acknowledgement on the part of TPM implementers, that
they can go further in terms of favouring the needs of implementing partners, and so creating a
more balanced relationship, which is crucial to the overall success of such projects.

Respondents were asked about accountability to communities and came up with the following
points.

10 WAYS DONORS CAN DEMOSTRATE ACCOUNTABILITY TO

COMMUNITIES THROUGH THE TPM PROCESS
TPM implementers were asked how they feel they can improve. They said:

1. Above all, think in terms of monitor safety, especially in conflict-affected areas, making sure that
sensitive data, such as the locations of targeted hospitals are never placed online unless in a
highly secure, regimented way.
2. Prioritise gender and social inclusion when discuss the methodology (including sampling) with
partners. It is challenging to reach vulnerable groups, but this has to be taken by all as a priority.
3. Ensuring that consent is gained and signed-for for any interview or action is essential. If monitors
use and carrying devices, this can be done on the screen or using the audio function.
4. Ensure that your TPM provider and your own team are aligned and robustly adopting best
practice with regards to conflict sensitivity. This should focus on monitors being alert to conflict
risks in the community, but also includes being alert to the conflict risks presented by the TPM
itself.
5. Ensuring that any questionnaires are as short as possible is key. It can be useful for the donor
team to go through the questionnaire piloting process, putting themselves in the shoes of being a
beneficiary or community member.
6. Keeping sample sizes to a practical minimum is important because communities often feel
inundated by such interviews. Take advice from experts on sampling, but also use your own
instinct – statistical reliability is important, but so is the impact of your TPM on communities and
the time they dedicate to helping you.
7. Providing feedback on findings to communities where possible – this may or may not be wanted
or sensible (depending on the topic), and where it is wanted, it may require a little creativity,
such as finding a pre-existing meeting where the community comes together, at which to share

37
 findings verbally. A brave donor may consider in particular sharing learnings and decisions around
what it can do better in a given area. An abridged version of a report can also be considered.
8. Liaising with other donors to reduce the overall burden on communities is also worthy of
consideration. It is unlikely that cooperation of this kind will eliminate the need to go to a
particular community, but sharing data, and so knowing some general facts may allow you to
shorten questionnaire lengths, or to understand the attribution of your intervention.
9. Ensuring that GDPR practices are followed; put someone in charge – inside the TPM implementer
– of rigorously following GDPR.
10.Ensure that data is end-to-end encrypted wherever necessary. In some settings malicious actors
may benefit from accessing data such as the location of projects. Most TPM providers are not
using end-to-end encryption; while this is justifiable in certain settings in others it can create
unreasonable risk.

6.6 Technology
Technology can be used in a range of ways to assist TPM.

Data collection - devices

Data collection through devices became mainstream around a

decade ago. Such devices offer a wide range of benefits, in
particular faster interviews and more efficient processing of data “Embrace
(Dette et al, 2016). They also offer other functionality that donors technology
should be aware of and may want to discuss with their TPM
partners; wherever you
 Supporting the Quality Control (QC) process by using audio can, but be old-
clips to listen to and check at least parts of interviews
 Supporting the Quality Control (QC) process and evidencing
fashioned when
that the interviews are bona fide through geo-tagging the you have to”
location in which the interview took place.
 Improving reporting quality by using devices to take photos Implementer
of any pertinent positive or negative findings at the
location, or of verified assets.

Surveying through telecoms providers

Much of the costs and time included in larger-scale TPM relates to the deployment of monitors to
engage with beneficiaries in remote areas. A company called GeoPoll has established direct relations
with telecoms providers allowing the pre-targeting of community members, which may be useful for
broad-scale surveys at a fraction of the cost and time required by conventional surveys, albeit with
reduced reliability of sampling quality. Meanwhile the establishment of call centres for phone
surveys is now far cheaper that it was ten years ago, it was said.

Big data

‘Big Data’ may be defined as the secondary analysis of very large datasets. At present, its usage in
the field of TPM appears negligible, driven by factors such as interoperability (Price, 2018). However,
the influence of Big Data on MEL and TPM may be significant. This ‘real-world’ data, for example
from mobile phone usage, can provide telling insights into the actual social outcomes of
interventions. For example, it is easy to imagine phone usage being taken as a proxy in a given
geographical area for the impact of an infrastructure project. For now, donors may instead want to

38
encourage LAs to make sure that they make reasonable effort to triangulate their findings with other
available data sources.

Aerial options

At present, the use of Unmanned Aerial Vehicles (UAVs)

or drones and satellite technology is not established in
the TPM environment. Of note is that UAVs are illegal in
some countries, and can also be seen as antagonistic.

Possible applications for this are however emerging;

UAVs could be sent to undertake interviews, or more
likely, to visually verify assets.

Use of satellites, however, has started to play a role. One

expert in this field stated that there has been a rapid
increase in the availability of satellite-based data, and
that such data is increasingly used for contextual
analysis, such as in the example from Iraq, to the right.

In this case, a specialist imaging firm used satellite to

provide a donor with information of destruction around
sites where investment had been made.

Tracking devices

For verification activities, in particular with regards to high-value assets, the attachment of Radio
Frequency Identification (RFID) or similar devices can be considered in place of the deployment of
monitors.

39
7 Analysis and reporting
Reporting should ideally have been tried and tested before and
during Inception. Beyond the basics considerations of OECD-DAC and
Red Amber Green (RAG) ratings, it’s key to think about
contextualising and sharing of findings, i.e. to consider the use of
dashboards, and how potentially to collaborate and share with other
donors.
7.1 Options for reporting
While reporting will be very-much situation-specific,
respondents were able to express the following general “Getting data is
advice:
only half the battle;
 Think carefully about the balance of frequency vs.
depth; many reports are too thick and too old the challenge is
 Try to imagine or even draft the report as early in getting actionable
the process as possible – it can be highly illuminating
to put oneself through the process and can greatly intelligence”
enhance the ToR
Implementer
 Speak to colleagues in other regions first to see if
they have templates that you can use, or that help
you to understand better what you really want the reports to say
 Do include photos as these can drive credibility and interest, and make all the difference in
driving traction of the report with key stakeholders
 Think about the comparability of your data – are there other data sets that you would want
to compare to? How about disaggregation of data? Is showing age and gender breakdown
enough, or can more be done?
 Consider asking your agency for a ‘video-presentation’ so you can share the findings in a
more visceral way
 Try to retain consistency in the report writer used, insisting on one person for regional work,
for example.

7.2 RAG ratings

For certain kinds of TPM, in particular where IP performance is a main focus, respondents favoured
the use of RAG (Red, Amber, Green) ratings. This is a simple visual device that provides a powerful,
objective judgment of the performance of a programme. There is not as yet a standard definition for
these three categories, and the norm is that each RAG rating is defined in a way that is appropriate
to the individual project.

Figure 13: Typical RAG-rating definitions

Red Serious issues identified that could fundamentally impact programming

Amber Significant issues identified - but not severely impacting programming

Green Strong performance, negligible or no issues identified that require action

40
Respondents said that the key to using RAG-rating are that:

 The use of the RAG-rating approach be discussed with IPs in advance

 The definitions for each of the three colour codes are robust; mutually exclusive and
precise
 That IPs have the opportunity to challenge any RAG rating (or other form of scoring).

Annex 3 includes a structure for a standard TPM report which blends RAG rating and OECD-DAC
criteria.

7.3 OECD-DAC criteria

A compelling combination is to use RAG rating on each of the OECD-DAC criteria.

OECD-DAC CRITERIA
These six criteria have become a backbone of evaluation and are often used in TPM, regardless of it
being a form of monitoring (rather than evaluation) because they nonetheless provide a straight-
forward and meaningful perspective on what needs to change.

1. RELEVANCE: Is the intervention doing the right things?

2. COHERENCE: How well does the intervention fit?
3. EFFECTIVENESS: is the intervention achieving its objectives?
4. EFFICIENCY: How well are resources being used?
5. IMPACT: What difference does the intervention make?
6. SUSTAINABILITY: Will the benefits last?

While these six are an established and widely-used set of lenses on performance, donors typically
feel free to adapt these also to the circumstances. For example, value for money may be included.
Or, depending on the project, it may be considered useful to include as standard ‘cross-cutting’
elements such as gender and social inclusion or conflict sensitivity.

7.4 Online reporting dashboards

‘Dashboards’ and data visualisations are increasingly created and used to share the results of a
project (Corlazzoli, 2014). Such dashboards often take the form of showing RAG-rated scoring of a
visited site, with access to the site being designed in such a way that each IP only sees their locations
and reports. TPM practitioners feel that this increase in interest reflects these benefits:

 Increased engagement with stakeholders, especially senior stakeholders on the donor-side,

thanks to a more insightful and usable interface, and more recent data
 Ability to contextualise the data against the broader situation
 Opportunity to engage directly with IPs and the LA and DC in once space
 That said, not all experiences with dashboards have been positive.
 Where dashboards have been trailed as a means of sharing reports, editing and
commenting on files collaboratively can prove challenging
 It can be difficult to drive users to the site if the user journey away from and back to their
own network is not simple, and if there is any considerable lag between fieldwork and
uploading.

41
 Figure 14: Reporting dashboard example

7.5 Collaboration between donors

Donors and TPM implementers agree that there is a clear and fairly pressing opportunity for donors
to collaborate better with each other to undertake TPM. The principal reasons for this are:

 Efficiency and Value For Money (VFM) –

avoiding duplication “I know one
 Accountability to communities – donors and
TPM implementers alike are mindful of the implementer who is
burden they place on
 Accountability to IPs
getting really tired
because everyone is
Some reflection was put forward of a future situation in
which donors create an integrated TPM system which monitoring them,
allows donors to:
really … everyone
 Share their planned TPM activities
 Share standard TPM templates in order to
should just come
 Allow comparison between reports together.”
At the same time, different actors are conscious that Donor
there are real-word practicalities which make this
difficult. First of all, much of this data is sensitive, and
sharing it may require a very safe, potentially end-to-end TPM needs to stop
encrypted database. Secondly, there is the practicality of
shared cost.
being about value for
To this end, respondents view allowed the distillation of
our money, but
a ‘phased’ approach to collaboration for donors to everyone’s money.”
consider.
Implementer

42
 • In an effort to drive some efficiencies,
donors share, in a systematic way, whom
1. Awareness only they plan to monitor, when and where.
No explicit intention to share data
presumed.

• In addition to the above, an explicit

agreement to share reports
systematcially, albeit on certain
2. Partial integration conditions. Moreover, agreement to
meet and discuss areas of concern and
potentially to take collaborative action.

• A vision for the future that would entail

common tools (so as to allow
3. Full integration benchmarking), an integrated fieldwork
plan to avoid duplication, and even
shared costs.

43
ANNEXES
ANNEX 1 – Bibliography
Chaudhri, S., Cordes, K., & Miller, N. (2017). Humanitarian Programming and Monitoring in
Inaccessible Conflict Settings: A Literature Review. Health Cluster. World Health Organisation.
https://www.researchgate.net/profile/Nathan_Miller5/publication/316693277_Humanitarian_progr
amming_and_monitoring_in_inaccessible_conflict_settings_A_literature_review/links/590ceccfaca2
7 22d185c150b/Humanitarian-programming-and-monitoring-in-inaccessible-conflict-settings-
Aliterature-review.pdf
Corlazzoli, V. (2014). ICTs for Monitoring and Evaluation of Peacebuilding Programmes. DFID:
Department for International Development. https://www.sfcg.org/wp-
content/uploads/2014/05/CCVRI-SSP-_ICT-and-ME-_Final.pdf
DCAF (2015) ‘Armed Non-State Actors: Current Trends & Future Challenges DCAF & Geneva Call’;
available online at: https://www.dcaf.ch/sites/default/files/publications/documents/ANSA_Final.pdf
Dette, R., Steets, J. & Sagmeister, E. (2016) Technologies for Monitoring in Insecure Environments.
Secure Access in Volatile Environments (SAVE) Toolkit. Global Public Policy Institute.
http://www.gppi.net/fileadmin/user_upload/media/pub/2016/SAVE__2016__Toolkit_on_Technolo
gi es_for_Monitoring_in_Insecure_Environments.pdf
Herbert, S. (2013). Remote management of projects in fragile states (GSDRC Helpdesk Research
Report 908) Birmingham, UK: GSDRC, University of Birmingham.
https://assets.publishing.service.gov.uk/media/57a089ffed915d3cfd00052a/hdq908.pdf
Kelly, L & Gaarder, M (2017) World Bank. https://ieg.worldbankgroup.org/blog/third-party-
monitoring-volatile-environments
Price, R (2018) ‘Approaches to Remote Monitoring in Fragile States’; available online at
https://gsdrc.org/wp-content/uploads/2018/01/1420-Remote-monitoring-in-fragile-states.pdf
Rivas, A., Guillemois, D., Rzeszut, K., and Lineker, B. (2015). Cross Cutting Evaluation of DFID’s
Approach to Remote Management in Somalia and North-East Kenya, Evaluation Report, DFID:
Department for International Development, London: Integrity Research and Consultancy.
https://www.gov.uk/government/publications/cross-cutting-evaluation-of-dfids-approach-to-
remote-management-in-somalia-and-north-east-kenya
Sagmeister, E. & Steets, J. with Derzsi-Horváth, A., & Hennion, C. (2016). The use of third-party
monitoring in insecure contexts: Lessons from Afghanistan, Somalia and Syria. Resource Paper from
the Secure Access in Volatile Environments (SAVE) research programme.
http://www.gppi.net/fileadmin/user_upload/media/pub/2016/SAVE__2016__The_use_of_thirdpart
y_monitoring_in_insecure_contexts.pdf
Taptue, A.M. & Hoogeveen, J. (2017, November 2). Project monitoring in fragile places does not
have to be expensive. World Bank blog post. https://blogs.worldbank.org/nasikiliza/project-
monitoring-in-fragile-places-does-not-have-to-be-expensive
United Nations, Third Party And Collaborative Monitoring (2015),
https://www.alnap.org/system/files/content/resource/files/main/third-party-and-collaborative-
monitoring-pv1.pdf
van Beijnum, M, van den Berg, W and van Veen, E (2018). Between a rock and a hard place;
Monitoring aid implementation in situations of conflict. Clingendael Institute.
https://www.clingendael.org/pub/2018/between-a-rock-and-a-hard-place/

44
ANNEX 2 – List of contributors
Name Organisation
Bruno Kessler Altai
Dhanya Williams Altai
Eric Davin Altai
Jeremie Toubkiss Altai
Justine Rubira Altai
Kamran Parwana Altai
Giorgio Saad Aktek
Dominic d'Angelo BDO UK
André Kahlmeyer CNC
Ribotaan Roy Coffey
Clare Winton DFID
Cyril Perus ECHO
Pedro Luis Rojo Garcia ECHO
Olivier Rousselle ECHO
Justin Ormand Ecorys
Cecile Delhez EU DEVCO
Milena Isakovic Suni EU DEVCO
Marcia Kammitsi EU FPI
Lea Tries EU FPI
David Bouanchaud EU FPI
Aminata Mar Thiem EU FPI
Marie-Luise Schwarzenberg EU FPI
Cedric Pierard EU FPI
Janine Abou Azzam EU FPI
Helga Pender EU FPI
Andy McLean First Call Partners
Michael Shaw Independent Consultant
An Hutton Independent Consultant
Kathryn Rzeszut Integrity International
Tom Gillhespy ITAD
Lameck Odallo Kimetrica
Philibert de Mercy Masae
Gilles Morain Masae
Gretchen Severson Burnham Global
Johnny Heald ORB
Sonya Schmidt Palladium
Victor Henriette Particip
Cecile Collin Particip
Marwa Bouka RMTeam
Bassam Al-Kuwatli RMTeam
Dr Althea-Maria Rivas University of Sussex
Travis Mayo USAID's Bureau for Policy, Planning and Learning

45
ANNEX 3 – Exemplar report structure
The structure below is one that is suitable for a TPM report describing the findings from a visit to an
implementation location in order to assess performance.

Section Section Main purpose Other information

# name

1 Context To provide the reader This section should provide the necessary
with the background background on: (i) the rationale for the TPM
they need to make itself (ii) the environment in and around the
best use of the report. location (maps may be useful here) including the
security situation and role of any key actors, (iii)
the aims of the IPs work (iv) any challenges that
the IP has experienced (v) a summary of any
existing MEL data, and (vi) the objectives of the
visit, assuming already agreed.

2 Executive A summary of the It is recommended to separate the summary

summary main findings, (specific, evidenced findings) from conclusions
conclusions and (the inference from those factual findings) from
recommendations. the recommendations (what should be done in
the light of the summary and conclusions.

Experience suggests that this section benefits

from being no longer than 2 pages, and that
recommendations are kept focussed, and that
there is an indication of the perceived
importance of each recommendation.

3 Main To provide the detail Typically this will follow the OECD-DAC structure
findings on what happened (relevance, coherence, effectiveness, efficiency,
during the visit. impact and sustainability), adapted as the donor
sees fit.

One option is for the TPM implementer to

describe separately the case for, and then
against, the IP having delivered on each OECD-
DAC element.

4 Annexes To supply any other This should include the tool/s (discussion guide,
pertinent additional questionnaire etc.) used and any other materials
information. that may be pertinent, such as (i) relevant news
article/s (ii) names of interviewees, if consent
was given, and (iii) photographic evidence
supporting contentions in the report.

46