DRAFT

Guide to conducting
reviews of organizations
s u p p l y i n g M & E t ra i n i n g
By:

Marie-Hlne Adrien

May 2001
Report prepared for: Operations
Evaluation Department (OED),
The World Bank, Washington,
D.C.

Conducting Reviews of M&E Training Organizations

Contents
1. Introduction

2. Conducting an Organizational Review

2.1
2.2
2.3
2.4
2.5

Overview
Planning the Review
Data Collection
Analysis
Report Writing

3
3
6
8
8

Appendices
Appendix I: Example of Terms of Reference for the Review
9
Appendix II: List of Data to Request in Advance
11
Appendix III: Sample Questions for Interviews/Focus Groups with Internal Staff 12
Appendix IV: Sample Questions for Interviews with External Stakeholders
14

June 2001

Conducting Reviews of M&E Training Organizations

1.

Introduction

As part of its efforts to promote sound governance the World Bank is working with a number of
countries to strengthen their monitoring and evaluation (M&E) capacities. This capacity-building
focuses on the ability of governments and civil society to conduct good-quality M&E and to use it
systematically in reviews of government performance. A key activity in strengthening capacities is
the provision of training to civil servants and others in a range of M&E tools and techniques,
including:
poverty monitoring, logical framework, performance monitoring indicators, results-based
management, program evaluation, impact evaluation, cost-benefit analysis, sector
reviews, policy analysis, rapid review techniques, participatory/stakeholder evaluation,
statistical sampling, research methodology, and development of M&E systems.
Training in M&E is ideally provided by local organizations such as universities, research centers,
specialist organizations such as civil service colleges or even NGOs. Another approach to
provision of M&E training is reliance on expatriate experts or foreign training organizations
however, this approach is judged less likely to lead to the development of sustainable local
capacity, nor is this approach well suited to adaptation of M&E training to national conditions and
experience. For these reasons, the Bank has provided trainer-training to staff of local organizations
in a number of sub-Saharan African and Central Asian countries, so that the capacity of these
organizations to deliver M&E training will be strengthened in a sustainable manner.
A number of governments put a high priority on the development of national capacities for M&E.
In any country, it is likely that training in some types of M&E is already provided. It is useful for
governments and the donors which support them, as well as for the organizations themselves, to
have a clear understanding of the quality of this training and also of the types of M&E in which
training is not currently offered locally.
This Guide was prepared by Universalia, a Canadian consulting company, in 2001, under contract
to the Operations Evaluation Department (OED) of the World Bank. Funding support for this work,
and also for the review discussed below, was provided by the Evaluation Office of the Ministry of
Foreign Affairs of the Netherlands Government. The author of the Guide was Marie-Hlne
Adrien, and the OED task manager was Keith Mackay.
The Guide was prepared in tandem with a review of training organizations in Ghana, also
conducted by Universalia under contract to OED. The Guide is intended to assist in the in-depth
review of organizations in developing countries which deliver training in M&E. As a number of
these organizations also provide consulting services in M&E, and are thus a potentially important
source of expertise for ministries and other agencies which wish to commission evaluations, the
Guide also encompasses an assessment of the ability of these training organizations to provide
M&E consulting services.
The Guide provides a general checklist of issues to be considered in conducting a review, which
can and should be adapted to the specific circumstances of the country.
The Guide covers the following issues:

the extent of supply of evaluation training, including the strengths and weaknesses in the
existing curriculum, and the supply of different areas of M&E consultancy work

adequacy of training facilities

staff experience and expertise in M&E

June 2001

Conducting Reviews of M&E Training Organizations

the organizations clientele and their feedback on the quality of M&E activities delivered
by the organization

the entrepreneurial and marketing ability of the organization to commercialize M&E

activities

external stakeholder feedback and support.

Charges for various M&E training courses.

June 2001

Conducting Reviews of M&E Training Organizations

2.

Conducting an Organizational Review

2.1

Overview

There are five basic steps in an organizational review:

1) Planning
2) Data collection
3) Data analysis
4) Report writing
5) Options and recommendations for action

2.2

Planning the Review

Which organizations can or could potentially supply M&E training and consultancy
services?
The range of organizations supplying M&E training and consultancy services in evaluation vary
from one country to the next. In general, however, the following organizations are often good
sources of these services:1

Universities (e.g. departments of administration, planning, or statistics)

Research Centers

Specialized agencies (civil service colleges, national audit offices, national statistical
agencies)

NGOs and other groups in Civil Society (evaluation associations, networks of evaluators)

Prior to a formal targeting of organizations for review, an M&E organizational capacity mapping
exercise can be conducted to ensure that all organizations that can or could supply M&E training
and consultancy services are considered.

This list excludes ministries, which might contain M&E departments or units with skilled staff in M&E, but
which usually would not offer their services to others outside their ministry.

June 2001

Conducting Reviews of M&E Training Organizations

What framework can be used to assess an organizations present or future abilities to
supply training and consultancy services in M&E?
A comprehensive framework for assessing the ability of an organization to supply M&E training
and consultancy services in M&E can be used to develop a checklist of issues.
This performance framework2 (Figure 1) presents three dimensions that can be assessed to
determine the organizations existing and potential of supply of M&E training and consultancy
services:
1. The external environment within which the organization is evolving, and its effects on the
organizations ability to supply M&E training and consultancy services. Key issues include the
level of external demand for M&E, stakeholder support for the organization as a delivery
agency, and the sanctions that exist for poor managerial performance. This review is also
referred to as a diagnosis of the public sector environment, including incentives and ethics.3

Figure 1: Framework for Conducting an Review of an Organization Supplying

M&E Training and Consultancy Services4

E x te r n a l E n v iro n m e n t

O r g a n iz a tio n a l
M o ti v a tio n fo r M & E
S u p p ly
H is to r y
M is s io n
C u ltu re
In c e n tiv e s R e w a r d s

A d m in is tra tiv e
P o litic a l
S o c ia l C u ltu r a l
T e c h n o lo g ic a l
E c o n o m ic
S ta k e h o ld e r

P e rfo r m a n c e in
M & E S u p p ly

O r g a n iz a tio n a l
C a p a c ity fo r M & E S u p p ly
L e a d e rs h ip
S tru c tu re
H u m a n re s o u rc e s
F in a n c ia l s y s te m s
P ro g r a m s e rv ic e s in M R E
In fra s tru c tu r e
E v a lu a tio n p r o c e s s e s
In te r- o rg a n iz a tio n a l
lin k a g e s
M a rk e tin g

Adapted from the Universalia / IDRC framework on assessing organizational performance. Enhancing
Organizational Performance: A toolbox for self-review. Lusthaus, C., Adrien, M.H., Anderson, G., Carden,
F. (1999) IDRC Publications.
3

See Evaluation Capacity Development: A Diagnostic Guide and Action Framework. Keith Mackay, World
Bank Operations Evaluation Department, ECD Working Paper Series no. 6, January 1999, p. 6-7.
4

Adapted from: Lusthaus, C., Adrien, M.H., Anderson, G., Carden, F. (1999) Enhancing Organizational
Performance. A toolbox for self-review. Ottawa, IDRC.

June 2001

Conducting Reviews of M&E Training Organizations

2. The organizational motivation for supplying M&E training and consultancy services, that is the
extent to which the mission and mandate of the organization support the supply of M&E
training and consultancy services, whether the organizational culture5 supports M&E for
accountability and learning purposes, and whether there are internal incentives for supplying
M&E training and consultancy services.
3. The organizational capacity of the organization, including leadership for supplying M&E,
infrastructure, faculty members, reach to other organizations, curriculum content,
administrative systems, and organizational structure for the delivery of M&E training and
consultancy services.
Based on this framework, specific Terms of Reference can be developed to guide the review.
Appendix I presents an example of Terms of Reference in which the emphasis was the review of
the M&E capacities of the organization, as well as stakeholder perspectives.

What should be requested from the organization, prior to the review?

Conducting a review requires a time investment from the organization. First, the organization must
allocate time for meetings with designated staff. Second, the organization must collect a series of
documents that will be important for the review. As a rule of thumb, plan to give the organization
a 3-week advance notification to enable it to get ready for the review. Appendix III provides an
example of a list of things to request in advance from the organization.

How long will it take?

The scope and depth of an review will vary depending on its purpose the uses to which the
review might be put, and on the time allocated for it, the number of issues to be assessed, the size
of the organization, and the availability of documents and informants. For an in-depth assessment
of the type outlined in this review, allow two to three days of data collection per organization,
including document review, visit, and interviews. For some organizations, especially those which
are suspected to have very weak capacities, a short visit might be sufficient to rule them out from
further consideration (or perhaps to indicate that a more in-depth assessment of them is
warranted).

Note that for public sector agencies the distinction between the external, public sector culture and the
internal organizational culture is at best a blurred one. For ministries and other public sector agencies the
latter is likely to be strongly influenced by the former.

June 2001

Conducting Reviews of M&E Training Organizations

2.3

Data Collection

Data for an organizational review typically comes from three sources: people (inside and external
to the organization), documents, and observation of the facilities.

People inside the organization

During the review, data should be collected from the following groups or individuals:

The Director, CEO (or equivalent): a two-hour individual face-to-face meeting is usually
needed for this interview.

The Director of M&E programs (or equivalent)

Faculty/staff members involved in the supply of M&E training and consultancy services,
training or consulting. Depending on staff availability, this meeting can be conducted as a
focus group interview followed by individual meetings with key informants from the group
as/ if they emerge.

Appendix IV provides a set of questions from which interview guides can be developed and
adapted for each category.
In addition, if time permits and if judged appropriate, interviews or focus groups can be conducted
with other internal staff members, such as:

Librarian

Students

Director/Coordinator/Head of Business Development

Director/Head/Coordinator of Finance

People external to the organization

Stakeholders opinions are important. They reflect the extent to which there is external support for
the organization, the organizations reputation, and its reach inside and outside the country.
Stakeholder groups generally include one or more of the following:

Existing donors and funding agencies

Competitors (other organizations supplying M&E training and consultancy services in the
country)

Participants who have attended training, or the ministries, local governments or other
organizations which employ them6

Organizations that have received M&E consulting services

National government agencies which oversight public sector reform efforts

Any other external stakeholder deemed appropriate

Appendix V provides a list of questions that can be adapted to develop interview guides for
external stakeholders.

While it is easy to track employers, it is in practice very difficult to track former participants or their direct
supervisors. It is useful to ask the organization for a list of participants (with their contact information).

June 2001

Conducting Reviews of M&E Training Organizations

Documents
Documents are a rich source of information. To the extent possible, try to obtain the following
documents for your review:
Background documents on the organization, including commercial brochures, annual
reports, strategic plans, previous evaluations of the organization, donors and funding
agencies reports on the organization, mission statements
Program/ course documents both general and specific to M&E, such as: lists or outlines of
courses, lecture notes on M&E courses and services, bibliographies, etc.
M&E course and workshop materials: review exercises, theory presented
Recent publications, particularly as they relate to the M&E field
Course evaluation ratings by participants
Curricula vitae of faculty and staff
Consulting reports conducted in the field of M&E
Client lists (type of organization, as well as level of participants involved) for the services
provided in general and for the M&E training and constancy services in particular
Follow-up to the services delivered (e.g. tracer studies of former participants)
It is also useful to review other documents such as local newspapers that may include
advertisements for the organizations M&E training and consultancy services.
Pricing structure (rates charged for training or consultancy services in M&E)
Availability of documents is often a limitation in these reviews. In some instances the documents
do not exist, in other cases they are not readily available or are only partially complete.

Observation
Another source of data is on-site observation of the premises. As you visit the organization, collect
data that will allow you to answer the following questions:
How well located is the organization? Is it easily accessible to participants?
Does the organization have the basic infrastructure to run a training course access to
water and electricity at all times (e.g. does the organization have a generator when there
are electricity shortages?)
Is there adequate ventilation in the classrooms and/or air-conditioning?
How many classrooms can accommodate the training?
Does the organization have modern training equipment such as overhead projectors,
flipcharts, VCR, LCD projectors, computers?
Can the organization provide accommodation and meals for the participants? For how
many? Is it of suitable quality?
Is the organization equipped to provide distance-learning sessions?
What is the level of technology (e.g. access to the Internet, how many staff members have
and/or use computers?)
What have former participants said about the adequacy of the premises?
What are the strengths and weaknesses of the organizations facilities?
How do the facilities of the organization compare with those of their competitors?

June 2001

Conducting Reviews of M&E Training Organizations

2.4

Analysis

Once the data are collected, your document analysis, observation notes and interview data can be
reviewed in light of the following criteria:
The mission and mandate of the organization in particular the extent to which supply of
M&E training and consultancy services is a key component of the mission and mandate.
Areas of strength and weakness in M&E that is, the areas of M&E in which the faculty
provides training, conducts research or consulting assignments.7 Organizations may have
very focused areas of M&E supply expertise; some organizations may be stronger in
training than in consulting. Similarly, faculty experience and expertise in M&E can vary in
terms of academic experience (research, publication, teaching, conference presentations)
and practitioners experience (conducting M&E assignments)
The adequacy of the organizations facilities for the delivery of M&E workshops
The organizations existing clients for M&E courses for specific and/or general training
courses (depending on the data available). It is useful to separate clients by categories
(private, public sector or Civil Society) and to examine any trends in clientele for M&E
training or consulting services.
The marketing ability of the organization that is, its capacity to sell courses to an external
target audience.
The level of stakeholder support that is, the degree to which either internal or external
stakeholders rate favorably the ability of the organization to deliver M&E training or to
undertake consultancy work in M&E.
The comparative advantage of the organization in providing training and consultancy
services in M&E, particularly from a pricing perspective.

2.5

Report Writing

The report will incorporate the results of the review and can be presented as follows:
Abstract, including major findings, and key recommendations
Background to the assignment: mandate of the study, methodology, data sources, team of
consultants, and limitations to the study.
Presentation of the review:
Presentation of the organization, including mission and mandate in the supply of M&E
Strengths and weaknesses in M&E supply
Assessment of staff expertise in M&E
Assessment of the facilities
Assessment of the entrepreneurial abilities of the organization to commercialize its M&E
training and consultancy services
Synthesis of stakeholder support for the organization
Recommendations
Conclusion

Areas such as, for example, poverty monitoring, logical framework, performance monitoring indicators,
results-based management, program evaluation, impact evaluation, cost-benefit analysis, sector reviews,
policy analysis, rapid review techniques, participatory/stakeholder evaluation, statistical sampling, research
methodology, consulting in M&E, training in M&E, developing M&E systems

June 2001

Conducting Reviews of M&E Training Organizations

Appendix I: Example of Terms of Reference

for the Review8
The Terms of Reference for the Review of the National Training Organizations are to:
1) Identify the type and amount of training which the organization currently provides in M&E
and in related areas: poverty monitoring; log frame and performance monitoring
indicators; result-based management; program evaluation (including impact evaluation);
cost-benefit analysis; sector review and policy analysis; rapid review techniques;
participatory/stakeholder evaluation; other areas of social science research and
quantitative analysis
2) Identify the different types of audience for whom training is provided, especially those in
the public sector and the nature of the training whether it is introductory (awarenesstraining) or more advanced.
3) Assess the degree to which the organization systematically conducts self-evaluation of its
training, based on levels 1 to 4 of Kirkpatricks model.
4) Review any existing evaluations of training provided focusing, where possible, on
participants reactions on the extent of use by participants in their jobs (i.e. Levels 1-3)
5) Assess the technical quality of the training provided its content, coverage and rigor
including its pedagogical effectiveness
6) Assess the M&E skills and experience of faculty/staff based on their curricula vitae and
on evaluations, research reports and other reports to which they have made an identified
contribution.
7) Assess the quality of training facilities, including the extent of reliable access to computers
and distance learning facilities.
8) Assess the reach of the organization in different regions of the country.
9) Assess the entrepreneurial and marketing ability of the organization how eager is the
organization to provide training and consulting services on a fee-for-service basis? Does
the organization have a well-developed marketing plan?
10) Assess the extent to which the organization already collaborates with other organizations
in the provision of training.
11) Consult with key stakeholders and client organizations to ascertain their judgment
concerning the strengths and weaknesses of the organization. These include government
central ministries, particularly those involved in provision of training for civil servants;
sector ministries that have specialist M&E units; in-country donor representatives; donor
headquarters (country and sector operational areas and evaluation offices); regional
development banks (especially their evaluation offices); other institutions, such as a
national evaluation association.
12) Prepare a report incorporating the above analysis, and including:
Identification of organizations strengths and weaknesses in M&E particularly their
ability to provide quality training in M&E, and
the ability of the organizations staff to undertake consulting work for government, civil
society and the private sector.
8

These ToRs are closely based on those prepared by the Operations Evaluation Department of the World
Bank for the Ghana review of organizations delivering M&E training.

June 2001

Conducting Reviews of M&E Training Organizations

13) Identify options to address substantive weaknesses. These might include provision of
trainer training in specific topics, or the provision of twinning/mentoring support.

10

June 2001

Conducting Reviews of M&E Training Organizations

Appendix II: List of Data to Request in

Advance
Planning the review: What to request from the organization being assessed
Background information about the organization such as the mission statement, strategic plan,
marketing plan, program brochures
1) List of training courses in areas related to Monitoring and Evaluation (M&E) such as, but
not restricted to: logframe, performance monitoring indicators, result-based-management;
program evaluation; cost-benefit analysis; rapid review techniques, qualitative and
quantitative research; questionnaire design. For each of these courses, The organization
should indicate:

The level at which the course is being given (beginners, intermediate, advance,

The duration of the course (is it part of an ongoing degree program or is it a professional
development course

How often the course has been given and when was the last time it was given

The name and credential of the trainers/teachers

The target audience and the actual types of participant (positions, organizations they come
from, demographic characteristics). . Of particular interest is the breakdown between
public sector participants (list the specific ministries or agencies), private sector and civil
society.

2) Course evaluation completed by participants i.e., participant feedback sheets

3) Results of tests or examinations of extent of participant learning
4) Follow-up evaluations9 of the training carried out by the organization (i.e. including tracer
studies, reports)
5) All training material that support the M&E courses, including: syllabus or outline,
bibliographies, reading lists, training content material etc.
6) Curriculum vitae of each Faculty member providing M&E training in the organization
7) M&E consultancy reports prepared by Faculty members of the organization
8) Links or alliances with other organizations providing M&E training
9) Pricing structures (rates charged ) for training or consultancy services in M&E.

Evaluation of the participants application of the training in the workplace , once the training is completed.

June 2001

11

Conducting Reviews of M&E Training Organizations

Appendix III: Sample Questions for

Interviews/Focus Groups with Internal Staff
Interview protocols can be developed and adapted for interviews with internal staff, using a
combination of the following questions:

What is the mission/mandate of your organization, in general and as it relates to the supply
of M&E?

How does your organization support the governments agenda on M&E?

What are the strengths of your organization in the supply of M&E training and consultancy
services?

What specific M&E training and consultancy services do you supply?10

How is the supply of M&E training and consultancy services integrated into your strategic
plan?

For how long has the organization been supplying M&E training and consultancy services?

Which are your main competitors?

In which types of M&E is your organization recognized as a supplier of training and

consultancy services? What is its reputation in these areas?

What segments/shares of the M&E market does your organization currently supply?

Has your share of the M&E supply market grown, declined, remained the same in the last
five years? How do you explain these trends?

What level of experience does the staff have in the supply of M&E both theoretical
(teaching, training) and practical (conducting M&E consulting assignments)?

In what areas of M&E have your staff specialized?11

Does your organization market/commercialize its activities? How?

What does your organization charge for different M&E courses? What are the range of
charges for its consultancy services? Are you competitive?

Are the M&E training and consultancy services of your organization price-competitive?

How financially autonomous is the organization today? What have been the trends in
financial autonomy in the last five years? Why?

What incentives and rewards exist in the organization to supply M&E training and
consultancy services?

What do you consider to be the strengths and weaknesses of your organization in the
supply of M&E training and consultancy services?

10

Areas such as, for example, poverty monitoring, logical framework, performance monitoring indicators,
results-based management, program evaluation, impact evaluation, cost-benefit analysis, sector reviews,
policy analysis, rapid review techniques, participatory/stakeholder evaluation, statistical sampling, research
methodology, consultancy in M&E, training in M&E, developing M&E systems.
11

12

See preceding note

June 2001

Conducting Reviews of M&E Training Organizations

With which organization/individuals/groups does your organization link/network in the

areas of M&E? What is the nature of these linkages?

For your organization, what are the main obstacles in supplying M&E training and
consultancy services?12

For your organization, what are the main opportunities in providing M&E training and
consultancy services?

What would your organization need to be able to become a better supplier of M&E
training and consultancy services?

Is there any specific support your organization needs to strengthen its M&E training and/or
its consultancy services?

12

Such as, for example, lack of demand for your services; charges and fees viewed as being too high;
political and economical constraints; technological difficulties; lack of stakeholder support; lack of
incentives for supplying M&E training and consultancy services; poor infrastructure, staff, systems,
leadership; funding; linkages; organizational culture that supports M&E.
June 2001

13

Conducting Reviews of M&E Training Organizations

Appendix IV: Sample Questions for

Interviews with External Stakeholders
Interview protocols can be developed and adapted for each external stakeholder, using a
combination of the following questions:

14

What is the nature of your relationship with the organization?

What kind of M&E training and consultancy services (training, consultancy work, other)
has the organization supplied to you?

How would you characterize the quality of the M&E training and consultancy services
provides to you by the organization?

From your perspective, is the organization a national or regional leader in the supply of
M&E training and consultancy services?

What lessons have you learned from working with this organization?

In comparison to other similar organizations supplying M&E training and consultancy

services in the country/region/municipality, how would you rate this organization?

Does the organization reach to various parts of the country/region?

How would you characterize the reputation of the organization in the country?

What are the main strengths and weaknesses of the organization?

How would you characterize the level of support that this organization has from the
national government?

What other organizations compare favorably/not favorably with this organization in the
supply of M&E training and consultancy services?

To what extent is the organization dependant on external funding?

Does the organization network/link with other organizations in the country?

Would you support the idea of strengthening the M&E capacities of the organization? In
what topics? Why?

Are the prices charged for training or consultancy services in M&E competitive?

June 2001