Independent Evaluation Policy

Approved by the BSTDB Board of Directors, 5 July 2023

1 PURPOSE & SCOPE…………………………………………………………………………………………….. 3

1.1 DEFINITION OF EVALUATION……………………………………………………...……………………………3

2 TERMS & ABBREVIATIONS …………………………………………………………………………………….4

2.1 TERMS……………………………………………………………………………………,,,…………………..4
2.2 ABBREVIATIONS………………………………………………………………………………………………...6

3 ROLES AND RESPONSIBILITIES………………………………………………………………………………7

3.1 THE OFFICE OF INDEPENDENT EVALUATION (OIE)…………………………………………………………….7
3.2 THE BOARD OF DIRECTORS…………………………………………………………………………………….8
3.3 THE AUDIT COMMITTEE OF THE BOARD OF DIRECTORS………………………………………………………8
3.4 THE PRESIDENT……………………………………………………………………………………….………..9
3.5 THE SENIOR MANAGEMENT…………………………………………………………………………….………9

4 POLICY……………………………………………………………………………………………………………10
4.1 FUNCTIONS OF EVALUATION………………………………………………………………………………….10
4.1.1 Definition ................................................................................................................................... 10
4.1.2 Objectives of Evaluation ........................................................................................................... 10
4.1.3 Independence, Transparency and Accountability .................................................................... 10
4.2 ACTIVITIES OF EVALUATION…………………………………………………………………………………..11
4.2.1 Timing ....................................................................................................................................... 11
4.2.2 The Operations Cycle and the Evaluation Function ................................................................. 12
4.2.3 Key Evaluation Stages and Activities ....................................................................................... 13
4.2.4 Evaluation and Strategies, Programs and Policies .................................................................. 13
4.2.5 Evaluation and Operations Quality at Entry .............................................................................. 14
4.2.6 Evaluation and Operations Implementation .............................................................................. 14
4.2.7 Evaluation and Operations Completion .................................................................................... 14
4.2.8 Workout Operations .................................................................................................................. 14
4.3 EVALUATION PRODUCTS AND THEIR DISSEMINATION………………………………………………………..15
4.3.1 Products .................................................................................................................................... 15
4.3.2 Dissemination ........................................................................................................................... 15

Independent Evaluation Policy Page 1 of 22

 4.3.2.1 Evaluation Products and Reports ......................................................................................... 15
4.3.2.2 Lessons Learned .................................................................................................................. 16
4.3.2.3 Raising Awareness of Independent Evaluations .................................................................. 16
4.4 EVALUATION METHODOLOGY ……………………………………………………………………….……17
4.4.1 Credible Purposeful Coverage.................................................................................................. 17
4.4.2 Sound Rating System ............................................................................................................... 17
4.4.3 Evaluation Rating Criteria ......................................................................................................... 17
4.4.4 Methodological Principles and Techniques .............................................................................. 18
4.4.4.1 Principles .............................................................................................................................. 18
4.4.4.2 Core Evaluation Techniques................................................................................................. 19
4.4.5 Reconciliation of Opinions ........................................................................................................ 20
4.4.6 Cooperation with Peers ............................................................................................................ 20

5 REFERENCES……………………………………………………………………………………………………21

6 ASSOCIATED DOCUMENTS…………………………………………………………………………………..21
6.1 LIST OF RECORDS & FORMS…………………………………………………………………………………21

7 DISTRIBUTION LIST…………………………………………………………………………………………….22

Independent Evaluation Policy Page 2 of 22

1 Purpose & Scope

This document is a Policy defining the independent evaluation of BSTDB’s operations, activities, strategies and
use of funds, that provides accountability and learning with the objective to improve overall institutional
performance and mandate fulfilment.

The Evaluation Policy governs the independent evaluation of operations, strategies, and activities within the
Bank with a focus on scope and objectives. It outlines the evaluation-related activities and responsibilities of
the Office of Independent Evaluation (OIE), its interaction with operation teams, senior management, the Board
of Directors (BoD) and any subordinate bodies designated by the BoD. It sets out the principles guiding
evaluation in the Bank and the specific internal roles and responsibilities required to accomplish and use
effective independent evaluation.

The Policy also provides the essential framework for procedures and methods, such as response by senior
management to evaluation findings, access to information, utilization of findings, internal circulation, and
external disclosure. In addition, it makes specific provisions pertaining to the Head of OIE, work program,
budgeting, and staff.

The scope of the Evaluation Policy is focused on OIE and its interaction with operation teams, the senior
management, the BoD, and the Board of Governors (BoG). This Policy defines the independent evaluation
function, performed by OIE, distinctly from operations and other functions, with specific duties to and oversight
by the BoD and its Audit Committee (AC), as well as the BoD and the President in his/her role as a chair of the
BoD. Through emphasis on two key principles - accountability and learning - the policy represents one of the
pillars of the institutional integrity and quality improvement of BSTDB.

The Evaluation Policy and system must stay abreast with prevailing best practice and applicable international
standards to maintain and enhance the Bank's operational quality as well as its public reputation, long-term
credibility, and ability to sustain the support of its shareholders, co-financiers, and investors as a first-class
development finance institution. For this purpose, the evaluation at the Bank is maintained in line with the “Good
Practice Standards” regularly updated by the Evaluation Cooperation Group (ECG) of the Multilateral
Development Banks, of which OIE is a member.

1.1 Definition of Evaluation

Evaluation at BSTDB is defined as the process of assessing operations, programs, activities, and strategies
through systematic and rigorous analysis of their outputs, outcomes and impact against expected results and
the overall mission of the Bank. BSTDB is a publicly owned institution, with a unique mandate. The support of
the Bank’s shareholders and partners requires objective and transparent demonstration of mandate fulfilment.
Effective, independent, and credible evaluation is key to accomplish this accountability. The objective,
evidence-based analysis of institutional performance provided by independent evaluation is the main pillar of
the wider framework of accountability within the Bank. The goal is to contribute to improved institutional
performance, relevance and reputation.

BSTDB’s accountability has several dimensions as the Bank must demonstrate delivery on a mandate with
multiple aspects of socio-economic development and cooperation within and among the countries of operation,
based on the guiding principles of sound banking and sustainability. This requires coherence and synergy in
the use of financing instruments, as well as a timely absorption of the lessons of experience within new activities
to ensure improved performance. To this end, independent evaluation has a central role in reinforcing
institutional accountability for the achievement of results; and, by providing objective analysis and relevant
findings to inform operational decisions for improved performance. It fulfils that role by providing credible
analysis, and independent judgment, along with evidence-based insights and recommendations. This facilitates
the senior management to deliver value and results, as well as the Boards to exercise their exclusive
responsibility of direction and oversight.

Independent Evaluation Policy Page 3 of 22

2 Terms & Abbreviations

2.1 Terms

The following terms are used in this Policy as respectively defined below:

Board Document: The entire set of operation information that is forwarded to the BoD for approval;

BSTDB's investment: The BSTDB’s financing instrument that is specific to the operation being evaluated:
mainly loans, loan guarantees and equity investments;

Client: Generally, the legal entity owning and implementing the project, and the BSTDB’s
investment counterpart;

Disclosure: The systematic distribution of evaluation findings through various media to the public,
in accordance with the BoD approved Public Information Policy;

Dissemination: The systematic distribution of evaluation findings through various media within the
BSTDB, without restriction as to its contents, with the aim of promoting awareness of
lessons for improved results;

Evaluation Staff: Staff of the Office (OIE);

Evaluator: Evaluation Staff or a consultant/collaborator assigned by OIE to take part in

evaluations;

Early Operating
Maturity: a) for non-financial sector operations: the year during which the substantially
completed operation will have generated at least 18 months of operating revenues for
the client; b) in the case of financial sector operations: the project year when most sub-
projects financed by the intermediary have reached at least 18 months past final
disbursement of sub-loans (or sub-investments in the case of equity fund projects);

Final Review
Document (FRD): The set of information / review regarding an operation to be financed, forwarded for
approval to the BoD (see Board Document above);

Independent
Evaluation: Evaluations undertaken by OIE, including Operation Performance Evaluation Reports,
Annual Evaluation Review Reports, Evaluation Studies, etc.;

Operation: The BSTDB’s objectives, activities and results in making and administering its
investment as partial financing of the Client's project;

Independent Evaluation Policy Page 4 of 22

Operation Completion
Report (OCR): A standard annual monitoring report for the year when the operation reaches early
operating maturity, with an attached evaluative addendum. The addendum is executed
according to a standard set of instructions prepared by OIE and featuring analysis of
specified performance dimensions with rated indicators, and lessons learned. OCRs
are executed on all "regular procedure" operations. OCR templates are prepared by
OIE;

Staff Directly Involved

with Operations: The staff involved in the identification, appraisal, implementation, supervision and
monitoring of BSTDB operations;

OCR Review/
Validation: The OIE process/instrument for conveying the findings of its desk review of each OCR;
scope addresses their quality (responsiveness to guidelines, depth and objectivity),
appropriateness of assigned performance ratings, and identified lessons;

Operation (or Program) Performance Evaluation

Report (OPER): Independent evaluation report prepared by the OIE on a sample of investment
operations; it is based on a thorough on-site assessment of use of funds and
addresses in depth the scope, analysis and ratings of the OCR;

Office of Independent
Evaluation (OIE): The BSTDB’s institutional unit charged with supporting and reviewing the products of
the self-assessment system for investment operations, in addition to its independent
evaluation studies, reports and other responsibilities;

Project/Operation: Generally, the Client’s capital project / operation or program and related business
activity that is partially financed by the BSTDB’s investment 1;

Self-evaluation: Evaluation of an investment operation (through an Operation Completion Report) that

is undertaken by the staff of the Operation Team which has day-to-day responsibility
for administering the investment being evaluated.

1
In the case of financial markets operations, generally refers to the financial intermediary’s lending or investment program that is partially
financed by the BSTDB.

Independent Evaluation Policy Page 5 of 22

2.2 Abbreviations

The following abbreviations are used in this Policy:

Abbreviation Full Wording of Abbreviation

AC Audit Committee (the BoD committee to which the Board delegated its oversight over OIE)

AEO Annual Evaluation Overview, a summary on OIE evaluation findings

BoD Board of Directors (BSTDB)

BoG Board of Governors (BSTDB)

BSTDB The Black Sea Trade and Development Bank, also referred to as the Bank

ECG Evaluation Cooperation Group, a standard setting organization of MDBs

FRD Final Review Document

IFI International Financial Institution

MDB Multilateral Development Bank

OCR Operation Completion Report

OIE Office of Independent Evaluation

OPER Operation Performance Evaluation Report, prepared by OIE

OPI Overall Performance Index, prepared by OIE based on 5 evaluation criteria

SWOT Strengths, Weaknesses, Opportunities and Threats, analytical concept

Independent Evaluation Policy Page 6 of 22

3 Roles and Responsibilities

Evaluation at the BSTDB involves the interaction of five authorities – the Office of Independent Evaluation
(OIE), BoD, AC, the President in his/her capacity as Chair of the BoD, and the senior management (Heads of
Divisions of BSTDB).

3.1 The Office of Independent Evaluation (OIE)

• OIE has the main responsibility for implementing and updating the evaluation policy and procedures,
in line with the applicable ECG International Standards.

• Reports functionally to the BoD through a Board Committee, the AC, and administratively to the
President in his/her capacity as Chair of the BoD.

• Provides agenda items and respective evaluation products such as reports, studies, work programs,
budgets, etc., to the AC, for discussion and information (budgets, work programs and specific
recommendations - for approval).

• Prepares and implements annual evaluation work programs and respective annual budgets; the OIE
work programs outline evaluation priorities, products, and activities. The OIE work program and budget
are submitted by OIE to the AC for review and approval. They address operational and staffing needs
that are fulfilled by OIE upon approval of the OIE budget by the AC. OIE’s dedicated budget is
ultimately approved by the BoD via the AC.

• Ensures that OIE activities, products and services address accountability and learning needs across
the spectrum of the Bank’s operational performance.

• Ensures the integrity of the BSTDB’s evaluation system by: developing methods and processes for
evaluation in partnership with peers and ECG in particular; validating and reviewing self-evaluations
(Operation Completion Reports) prepared by operation teams, and assessing the reliability of the
process; conducting independent evaluations, assessing any aspect of the Bank’s activity that affects
mandate fulfilment; and, reporting annually to the BoD and BoG on the performance of the evaluation
system and the results and findings it reveals.

• Provides the BoD and the AC with regular reports and studies, issued and approved by OIE, on
evaluation activities, findings, lessons learned and challenges within the Bank, through key vehicles
such as the Annual Evaluation Overviews, the latter also being disseminated through a dedicated
chapter in the Bank’s Annual Reports.

• Facilitates and promotes the use of evaluation findings and recommendations by disseminating them
internally and externally, as appropriate; and maintaining a stock of lessons learned, including those
from relevant peer institutions.

• Provides training and familiarization services on evaluation within the Bank to strengthen self-
evaluation and encourage the use of evaluation findings and lessons learned.

• Participates in external fora and membership institutions, to represent BSTDB’s views and interests
with respect to evaluation issues, to contribute expertise and experience to the development of
international standards, with a particular reference to ECG.

Independent Evaluation Policy Page 7 of 22

 • Participates in joint evaluation work and staff exchanges with partner organizations where appropriate,
based on potential or actual parallel or co-financing.

3.2 The Board of Directors

• Approves the Bank’s Evaluation Policy and the AC Terms of Reference and oversees their
implementation.

• Defines the terms of reference for, appoints, supervises, and assesses the head of OIE, in line with
the applicable international standards as maintained by ECG and delegated responsibilities to its the
AC, as outlined further under 3.3.

• Approves the OIE work program and budget as stand-alone OIE-specific documents, as submitted
(via the AC) by OIE.

• Discusses reports, studies and other outputs prepared, approved, and submitted by OIE and gives
direction on their recommendations.

• Delegates any of its evaluation-related responsibilities to a designated committee - the AC.

3.3 The Audit Committee of the Board of Directors

The roles and responsibilities of the AC to which the BoD delegates evaluation functions and oversight
of OIE are reflected in the terms of reference for that committee. It performs the delegated functions of
the Board of Directors regarding the oversight of and interaction with OIE, as per the adhered to
international (ECG) standards, the Statement of the BSTDB’s Audit Committee of the Board of
Directors, to ECG (during the 52nd meeting) and respective BoD decision BD-13-020-93-BD2013-049,
including:

• Approval of the OIE annual budget as clearly distinct and identifiable, not subject to reallocation
without BoD approval, together with the OIE annual work program, prepared and submitted by
OIE;

• Holding of discussions, upon own or OIE initiative, on relevant evaluation matters such as key
findings, studies, recommendations, resource needs, compliance with respective ECG
standards, challenges to credibility and independence, etc.

• Contributing to the performance appraisal review of the Head of OIE by providing feedback on
the implementation of the respective annual work program. Any recruitment/replacement of a
Head of OIE will include a representative of the AC at the selection/review panel, to reflect the
Committee's oversight/assessment role vis-a-vis the independent evaluation function at the
Bank, as per applicable ECG standards and this Policy.

• Reviewing and safeguarding the independence of OIE, assuring that relevant policies,
procedures, methodologies, budget, and staffing resources are adequate and up to international
standards. Recruitment of OIE staff should follow the Bank’s recruitment procedures, but the OIE
Head should have a decisive/approval role in the criteria and actual selection, in line with the
applicable international ECG standards that safeguard independence and prevention of conflict
of interest.

Independent Evaluation Policy Page 8 of 22

 • Review of senior management’s responses to evaluation report findings and recommendations
and the respective actions taken, in consultation with the Head of OIE.

• Oversees progress on agreed evaluation recommendations (by senior management, Board

Subordinate Committees or Boards) and periodically informs the BoD on respective
implementation.

3.4 The President

• Interacts with OIE in his/her capacity as BoD Chair with a focus on critical performance issues,
forwarding all evaluation agenda items and documents as/when submitted by OIE to the BoG, BoD,
and senior management, for information/action, without any amendments or clearances, as well as on
relevant administrative matters such as annual leave and business trips as per the budget and work
program of OIE (approved by the AC).

• Ensures adequate and effective engagement in evaluation-related matters before the BoD or Board
subordinate bodies, safeguarding and promoting the key principles of independence of evaluation.

• Interacts with OIE in his/her capacity as BoD Chair on relevant evaluation-related matters and
presentations, with a focus on safeguarding both the independence of evaluation and the function
from interference.

• Ensures that institutional processes, interactions and resources are adequate to accomplish
evaluation-related activities, in line with the OIE work program and budget as approved by the AC /
BoD

• Interacts with the Head of OIE on relevant strategic issues and their evaluation.

3.5 The Senior Management

• Takes part in relevant evaluation-related discussions at the Board or Board subordinate Committees.

• Ensures an effective system of self-evaluation (operation completion reports) and reports periodically
to the BoD on its scope and quality.

• Responds in writing to OIE reports and recommendations and informs the BoD on implementation of
agreed recommendations.

• Ensures that proposed operations clearly specify expected results and related ex-ante performance
indicators, to allow effective evaluation.

• Ensures that programs, policies, and strategies identify their expected results with sufficient specificity,
to allow effective evaluation.

• Ensures that relevant evaluation findings and lessons are adequately reflected in matters placed
before the BoD.

Independent Evaluation Policy Page 9 of 22

 • Ensures institutional processes and resources sufficient to accomplish evaluation-related activities, in
line with OIE budget and work program.

4 Policy

4.1 Functions of Evaluation

4.1.1 Definition

Evaluation at BSTDB is defined as the process of assessing operations, programs, activities, and strategies
through systematic and rigorous analysis of their outputs, outcomes and impact against expected results and
the overall mission of the Bank.

While evaluation is mainly focused on completed operations (ex-post-evaluation), OIE also performs mid-term
and real time evaluations in some cases, as well as a wide range of evaluation studies, based on needs and
available resources.

4.1.2 Objectives of Evaluation

The evaluation function has two interlinked objectives, serving the overall purpose of assessing mandate
fulfilment and drawing lessons for improved future operations:

a) Accountability – to reveal the results of the Bank’s operations/activities, both intended and otherwise,
and assess their contribution to the Bank’s mission.

b) Quality management improvement – to derive significant lessons learned from past experience in
view of using them for improving future operations, strategies and activities.

Evaluation provides a basis for rendering judgements, for the benefit of the Bank's staff, senior management,
Boards, and shareholders, concerning the relative success or failure of the Bank’s operations vis-à-vis the
BSTDB’s mandate.

The Evaluation Policy and OIE aim toward a continuous improvement of the BSTDB’s effectiveness, credibility,
and reputation, as well as to learn from an independent in-depth analysis. Therefore, the Bank relies on a key
feedback process that provides the decision-makers at different levels with lessons learned from both own
evaluations as well as other IFIs’ evaluations.

4.1.3 Independence, Transparency and Accountability

The evaluation system has two tiers: (i) self-assessment by the staff directly involved with operations, resulting
in Operation Completion Reports (OCR) and (ii) independent and rigorous evaluation performed by the OIE.

Because independence, equal ranking vis-à-vis other Bank units and objectivity are vital for the credibility and
prudence of the evaluation, the latter is performed and managed by a dedicated independent office, the OIE.
While integrated into the organizational structure of BSTDB, the OIE functions independently from operations
and related decision-making, according to this BoD-approved policy that specifies its role and key operating
principles. While some evaluation reports are presented to the Credit or Management Committees, as relevant,
the OIE reports to the BoD via a designated committee, the AC, which has an oversight role vis-à-vis the
evaluation function, program, and budget. In general, the OIE ensures the relevance, quality, and impartiality
of the BSTDB's evaluation system. To meet its responsibilities under this Policy, it has unrestricted access to

Independent Evaluation Policy Page 10 of 22

the BSTDB's staff, records, co-financiers, clients, and operations, with due respect to the Bank’s Policy on
Personal Data Protection. To this end it will perform the following steps, when interacting with external parties
for the purpose of an independent evaluation:

• Upon commencing an evaluation interaction with external parties, OIE will dully communicate the following
statement to them: “All the personal data provided herein shall be processed by BSTDB, in accordance
with its Policy on Personal Data Protection (https://www.bstdb.org/privacy), for the purpose of carrying out
its evaluation process and specifically for the assessment of the operations, programs, activities and
strategies of the Bank. The legal basis of such processing by the Bank is serving the legitimate interests
pursued by the Bank to assess its mandate fulfilment and draw lessons for improved future operations. In
case you have a query regarding the protection of your personal data held by BSTDB or if you wish to
exercise your rights as Data Subject, please contact BSTDB’s Data Protection Officer, at: dpo@bstdb.org

• When OIE will use surveys and if the collection of personal data is not required to achieve the survey
goals, the OIE should consider using anonymized surveys that do not lead to the identification of the
survey participants. Such surveys will include a relevant disclaimer such as “the survey information to be
provided will not be in a form that identifies any natural persons involved. Answers should be provided in
an anonymized manner.” In case personal data are disclosed nonetheless in an anonymized survey’s
answers, OIE will delete any such personal data upon obtaining them.

The evaluation normally commences after an operation is completed. In addition to operations ex-post-
evaluation, the OIE also conducts various evaluation studies, which focus on a particular problem, issue, sector,
country, programme, etc.

The evaluation process requires interaction among (i) the Bank’s staff directly involved with operations, (ii) the
clients and (iii) the Evaluator(s). A successful functioning of the evaluation system requires a clear-cut definition
of roles: clients provide information and cooperate in the evaluation process as per their obligations in the
respective legal documents; the staff directly involved with operations provide all available information, along
with an Operation Completion Report; and the OIE performs independent validation of self-evaluation and
evaluates in-depth the respective operation, program or activity through a rigorous internationally harmonized
methodology.

The transparency and independence of the evaluation system provide for timely, accurate and objective
generation of lessons learned, as well as prompt feedback of these lessons to all levels of governance. For this
reason, the evaluation function interacts with various units at arms-length, avoiding direct interference in the
decision-making process.

4.2 Activities of Evaluation

4.2.1 Timing

To be effective, the evaluation should have appropriate timing. Ex-post-evaluation requires a certain degree of
evaluability, known as Early Operating Maturity2, which may vary from case to case3. Depending on the
complexity and relevance of the given operation, some evaluations may be performed soon after operation’s

2
Usually, post-evaluation will commence after a period of at least 18 months of operating revenues for the Client (from the evaluated
operation) or at least 18 months past final disbursement of sub-loans (or sub-investments in the case of equity investments). See definition.

3
There is a recognized trade-off between delaying evaluation long enough to have a reliable basis for making updated projections for
outcome and impact judgments (justifies later evaluation) and reviewing results as early as possible for their lessons, relevance to evolving
corporate priorities and accountability uses (justifies earlier evaluation). As empirical evidence suggests that projections tend to be
optimistic, earlier evaluations will tend to reflect better outcomes than will true ex-post ratings.

Independent Evaluation Policy Page 11 of 22

completion/disbursement4, while other may be assessed at a later stage but normally well before full repayment.
The timing of Evaluation Studies and real time evaluations will be based on needs, demand, and resource
availability of the OIE.

4.2.2 The Operations Cycle and the Evaluation Function

The focus of the evaluation function is at the end of the Operations Cycle, i.e. at completion. However, through
its feedback system, the evaluation function affects indirectly the other phases of the Operations Cycle, as well
as some conceptual/programming activities.

OPERATION CYCLE AND EVALUATION CYCLE IN INTERACTION

THE OPERATIONS CYCLE THE EVALUATION FUNCTION

• Strategy/Programme* • Lessons learned

• Initiation (Concept Clearance) • Lessons learned

• Appraisal and Due Diligence* • Lessons learned

• Negotiations
EVALUATION
INTERACTION
• Approval and Effectiveness • Real Time Evaluation

• Implementation and Supervision* • Support to self-evaluation

• Completion** • Evaluation Reports/feedback

* Arm’s-length” interaction, based on demand and feedback of lessons

** Substantial interaction

The OIE provides, upon request, a variety of relevant lessons learned from internal and external (other IFIs)
evaluations. It also provides support to the process of self-assessment through guidance and templates for the
preparation of Operation Completion Reports.

4
It is assumed that final disbursement and physical completion (i.e. when the physical investments are in place and are operational) occur
at the same time. However, depending on the operation nature, the physical completion, when the facilities are operational, might occur
after the final disbursement. In this case, the evaluation process will use this later date as a starting point, rather than the date of final
disbursement.

Independent Evaluation Policy Page 12 of 22

4.2.3 Key Evaluation Stages and Activities

The evaluation process has three main stages: (i) self-assessment, also known as completion reporting; (ii)
independent evaluation, known as performance reporting, and; (iii) synthesis of evaluation findings also known
as Evaluation Overviews.

(i) Self-assessment

Self-assessment of the operation's performance is conducted by the staff directly involved with
operations and results in an Operation Completion Report (OCR). An OCR is produced once in the
lifetime of each operation. The due date typically falls after the operation’s technical completion
(typically soon after full disbursement but not later than at least 2 years prior to final repayment), upon
reaching Early Operating Maturity. The OCR represents the only comprehensive and systematic self-
assessment of an operation upon its completed implementation, which looks at performance in the light
of the Bank's mandate and reveals lessons learned. It is an important starting point of the independent
evaluation process.

(ii) Independent Evaluation

The completed operation, for which an OCR has been issued, is thoroughly and systematically
evaluated by the OIE. This results in various types of evaluation reports, depending on their depth,
called Operation Performance Evaluation Reports (OPER), or in aggregate evaluation reports such as
evaluation studies or annual evaluation overviews. When a full-fledged OPER is not produced, due to
a large portfolio of completed operations, the OIE will, at least, review the OCR to independently verify
the validity of its statements, and report the results, typically within annual evaluation overviews. This
ensures an independent validation of contents, lessons, and self-evaluation ratings. Apart from post-
evaluation upon completion, real time evaluations are also conducted by the OIE, based on importance
and data from monitoring reports and other sources. Evaluation of still disbursing operations may result
in analysis of any implementation problems or policy issues encountered and include insights that will
facilitate the decision-makers in their effort to ensure enhancement, steering, or restructuring.

(iii) Synthesis of Evaluation Findings and Evaluation Studies

In this stage, the OIE synthesises evaluation findings in annual, sector, country, and/or thematic-
focused reports. These analyse patterns of experience, distil lessons learned and provide suggestions
for systemic improvements. A typical form of synthesis evaluation is the Annual Evaluation Overview.
It is produced annually to compare essential evaluation findings and distil strategic lessons, based on
such comparisons. It contributes to the overall BSTDB accountability and public visibility, as
aggregated evaluation findings allow more open dissemination and comparisons with other IFIs. This
report is the key document for presenting to the BoD and BoG emerging patterns of experience that
may affect entire segments of the Bank’s portfolio of operations. Based on its findings, as well as
external (IFI) sources of relevant lessons, the OIE feeds experience, and lessons learned into the
operation cycle.

4.2.4 Evaluation and Strategies, Programs and Policies

OIE is not involved in the process of preparing strategies and programs but may provide relevant lessons
learned upon request. Typically, the evaluation of a completed strategy is issued as a stand-alone evaluation
report to facilitate the preparation of the next strategy. For this purpose, respective departments contact OIE to
obtain a review of past experiences and reflect it in the respective programme or strategy. When evaluating
operations, the evaluation staff verifies to what extent a particular operation follows the relevant current
strategies and programs, to issue a judgement on its relevance. To assess the effectiveness of programs and
strategies, and in view of suggesting measures for enhancement, the OIE evaluates their implementation and
reports the results to the BoD and BoG.

Independent Evaluation Policy Page 13 of 22

4.2.5 Evaluation and Operations Quality at Entry

During an operation’s design, preferably immediately after the Concept Clearance stage, the staff directly
involved with operations consult OIE to obtain and include in the operations documentation a list of relevant
lessons learned. This information becomes part of a dedicated section (in the FRD and the Board Document)
and enhances the quality at entry.

OIE, due to the independence of the evaluation function, is not directly or actively involved in the preparation
and implementation of any operation. Its involvement is limited to the provision of experience and lessons
learned (external and internal). For this reason, the OIE will maintain regular interaction with other IFIs and
respective evaluation groups, to collect and validate relevant lessons of experience across sectors and themes
of interest.

4.2.6 Evaluation and Operations Implementation

OIE, due to its independent position, is not involved during the operation implementation stage. It only provides
relevant lessons learned upon request from staff directly involved with operations. In selected cases, when the
implementation prospects of an operation are considered critically important, OIE performs a real time
evaluation, which may focus on a particular issue of concern.

4.2.7 Evaluation and Operations Completion

Staff directly involved with operations prepare an Operation Completion Report (OCR) as a replacement of the
forthcoming regular Supervision/Monitoring Report. The OCR is due within six months upon technical
completion of an operation, typically at least 2 years before final repayment. For partially or fully disbursed
investment operations, which are exited, cancelled, prepaid, or terminated for other reasons, an OCR is
prepared within three months thereafter. This is important since the Bank can learn a lot from operations that
could not be completed (because of unexpected problems) or those that implied an early exit or termination for
other reasons. When an operation has been cancelled before a disbursement has taken place, within three
months of the date of cancellation the Operation Leader prepare a Lessons Learned Memorandum that reveals
what has been learned from the experience and if there are any implications for future operations. The
Operation Leader sends a copy of this memorandum to OIE so that an evaluation can start as early as possible.

OIE develop and maintain, in consultation with other relevant units, the template of the OCR and provide it,
along with guidance/training on how to prepare the OCR, to the staff directly involved with operations. The
template is basically identical with the regular Monitoring Report template but contains an additional section
that addresses self-evaluation issues. The OCR becomes the basic self-evaluation record, ensuring the primary
accountability of operation’s performance. The OIE validates and rates the quality and objectiveness of those
OCRs, which concern operations not selected for subsequent full-fledged evaluation.

The core activity of the OIE is to evaluate the performance of completed operations, revealing their contribution
to BSTDB’s mission. The evaluation ratings are derived via internationally harmonised methodology and
represent the accountability function of evaluation while the section of lessons learned represents the learning
function. The evaluation findings are based on the OCR, all relevant files/documents, a site visit and respective
consultation with sponsors, clients, Bank staff and other relevant parties. When required by OIE, specialised
industry consultants will also support the evaluations.

4.2.8 Workout Operations

Workout operations (restructuring and corporate recovery) are among the best sources of important lessons
learned and their evaluation has proven to contribute a lot towards the performance of future operations. For
each operation that entered corporate recovery, a Lessons Learned Memorandum should be submitted to the
Evaluation Office by the team that undertakes corporate recovery. The timing of the evaluation of workout
operations is a sensitive issue, as evaluation may interfere with ongoing recovery work and thus increase the
cost of exit. Therefore, OIE will choose an appropriate timing, addressing this perspective.

Independent Evaluation Policy Page 14 of 22

4.3 Evaluation Products and their Dissemination

4.3.1 Products

The evaluation activities result in the production and dissemination of written documents that may take the form
of a Report, Study or contribution to the Annual Evaluation Overview. The main evaluation products are
Operation Performance Evaluation Reports and Annual Evaluation Overviews.

The OIE produces various relevant outputs through which it delivers its findings and recommendations on
issues that were subject of evaluation. These include the following:

• Evaluation Studies. Evaluation of the quality, effectiveness, and efficiency of implementing various
Bank Programs, Strategies and Policies results in Evaluation Studies. They provide independent in-
depth analysis and suggest amendments and measures to improve the Bank’s performance and
achievement of its mission. Some Evaluation Studies analyse and compare the findings of evaluations
of individual operations to reveal trends as well as to distil lessons that have implications beyond the
operation or sector. Other Evaluation Studies look at key issues of interest for the Bank and may focus
on international/IFI best practice, experience or lessons learned. Evaluation Studies may analyse
specific sector- or country- related issues, or both.

• Annual Evaluation Overviews. These reports aggregate and compare the findings of OIE on an annual
basis. Thus, an overall picture of performance by countries, sectors, and types of operations, will be
revealed for the attention of relevant staff, senior management and the BoD and BoG. These reports
do not contain commercially sensitive / operation specific information and therefore represent the main
vehicle for broader disclosure and accountability on the Bank’s overall performance and are also
included within the Bank’s Annual Reports.

4.3.2 Dissemination

4.3.2.1 Evaluation Products and Reports

Based on the type and role of the specific evaluation product, OIE may distribute it in draft form to selected
parties within and outside the Bank, as appropriate, as well as hold respective meetings internally or externally
(with clients and other stakeholders), to gather further information and comments, with a focus on evidence
and counterfactuals, based on applicable methodology.

The draft and final evaluation reports are issued by their author within OIE and transmitted to recipients through
the Head of OIE, without external clearances or amendments. If a recipient does not agree with the findings,
judgements or recommendations expressed in the evaluation report, and if differences of opinion cannot be
reconciled through discussion/evidence, the differing views will be noted/ mentioned in the final version which
is a sole responsibility of OIE.

Any draft evaluation product is finalised and cleared by the Head of OIE, who submits it for respective
dissemination of information as relevant, with key evaluation products being submitted to the BoD and BoG via
the President in his/her role as a BoD chair. The only evaluation products that are subject to external approvals
are the OIE work program and budget, as well as amendments to this policy, approved directly/solely by the
AC and the BoD respectively.

The evaluation products are made available to Bank staff by the OIE, with a focus on units addressed by or
relevant to the respective evaluation product. OIE holds relevant clarifications, awareness and training
regarding its products and the evaluation policy, at its discretion. It decides on the form, content and means of
internal and external dissemination, taking respective measures to protect commercial and personal data, as
per the applicable policies and evaluation standards.

Independent Evaluation Policy Page 15 of 22

Various evaluation products, as relevant, are distributed by OIE internally (to senior management/Credit
Committees, AC, Boards, staff) and externally (to partner networks, institutions, stakeholders), as appropriate.

4.3.2.2 Lessons Learned

The OIE distributes various other written documents on evaluation findings and lessons internally and
externally. It contributes relevant past-experience and lessons learned (internal and external) to the staff directly
involved with operations after Concept Clearance to enhance new operations and strategies. The OIE
maintains the Bank's lessons learned library. It provides guidance and training to relevant staff, to ensure
learning from experience, and institutional memory. Confidential case presentations are made intermittently to
Directors on the AC of the BoD upon their request or upon the initiative of OIE. Other recurrent dissemination
includes case reviews with the senior management/Credit Committees.

Experience has shown that achieving awareness and internalisation of lessons and their systematic application
in new operations is the weakest link in the evaluation learning cycle. It is the responsibility of Operation Leaders
to ensure that past lessons have been systematically researched, documented, and applied in new operations.
Standard processing documentation for new operations includes a section on relevant past lessons, as
provided by OIE, to be consulted during preparation and implementation of each operation. OIE periodically
reviews and reports annually to the BoD and BoG in its Annual Evaluation Overview on the quality of
responsiveness to provided lessons learned and recommendations.

4.3.2.3 Raising Awareness of Independent Evaluations

Targeted and regular dissemination of all evaluation outputs/reports is an essential activity that keeps the
evaluation feedback system in operation. Therefore, awareness of evaluations, as well as accompanying
dissemination measures, e.g., discussions and seminars, is offered by OIE to foster utilization and impact, with
a focus on key lessons learned. They target both new and current staff with the aim of gearing internal and
external lessons to improvements of operations. Such occasions often include appropriate joint
activities/workshops with partner IFIs.

Adequate performance of the accountability function of OIE and the importance of networking with other IFIs
require a continuous effort to ensure wider dissemination, outside the Bank, as appropriate. This implies the
need of preparing summaries, case studies and sanitised reports that are in line with the BSTDB’s Public
Information Policy and Policy on Protection of Personal Data and do not compromise sources or clients. To
ensure a high degree of connectivity with the development community, the OIE deploys a variety of means and
channels for dissemination, including a dedicated page on the BSTDB website. Connections with relevant
development forums and gateways should be maintained consistently.

To perform its quality management function, the OIE ensures the establishment, development and deployment
of a systematic and comprehensive knowledge base that serves the needs of both senior and operational
management5. This base will contain key lessons and policy implications. Awareness of the existence and
value of the evaluation knowledge base will be raised and maintained by the OIE with the support of the senior
management.

5
Unit heads and directors of BSTDB.

Independent Evaluation Policy Page 16 of 22

4.4 Evaluation Methodology

4.4.1 Credible Purposeful Coverage

OIE ensures adequate coverage of operations to be evaluated to achieve credibility. Given the importance of
higher-level evaluations, beyond operations, a purposeful sampling will complement a wide review/validation
of self-evaluation reports. Sample selection criteria are part of the methodology and include the following: new
sector/country/instrument, balanced sector/country coverage, potential for lessons and problem operations,
and a degree of randomness. The focus will be on operations and activities of particular interest.

4.4.2 Sound Rating System

A sound evaluation rating system is at the core of a prudent evaluation as it ensures accountability and
comparability. It is a standardized set of (evaluation) rating criteria and a rating scale of predefined categories.
The system includes (i) appropriately and clearly defined rating criteria and rating factors, as addressed in 4.4.3
below and (ii) an even number6 of at least four categories/scores on the rating scale.

The four rating categories have numerical expression that allows calculations/aggregation and form the basis
for all ratings:

➢ Category 1: (Excellent or equivalent; 4 scores): Most of the key objectives were achieved or it is highly
likely to achieve substantial development / cooperation results, without major shortcomings.

➢ Category 2: (Satisfactory or equivalent; 3 scores): Most of the key objectives were achieved and
satisfactory development / cooperation results were or are expected to be achieved with only a few
shortcomings.

➢ Category 3: (Partly Unsatisfactory or equivalent; 2 scores): Operation failed to achieve most of its key
objectives, has not yielded and is not expected to yield substantial development / cooperation results,
and has significant shortcomings.

➢ Category 4: (Unsatisfactory or equivalent; 1 score): Operation failed to achieve any of its key objectives
and is not expected to yield worthwhile development / cooperation results.

4.4.3 Evaluation Rating Criteria

The evaluation rating system is based on 5 core rating criteria, commonly used by the Multilateral Development
Banks. It also includes an aggregate rating indicator – the Overall Performance Index (OPI), derived from the
ratings on these 5 criteria. The OPI ensures that each operation will have an aggregated rating that allows for
further calculation of ratings for entire portfolios and sectors. It will also allow for comparisons with earlier
periods, as well as other IFIs’ performance. While these rating criteria and other elements of the evaluation
rating system are used for long periods as a matter of consistency and comparability, OIE reviews and updates
them as relevant, in line with respective international standards. The essence of the evaluation rating criteria
could be summarized as follows:

a) RELEVANCE: Consistency of operational objectives with the BSTDB overall cooperation and
development mission and relevant strategies/programs. It is based on BSTDB priorities as they stand
at the time of evaluation.

6
This is to require the evaluator to come to a positive or a negative judgment, just as operational decision makers are required to make a
“go” or “no-go” decision at the time of appraisal.

Independent Evaluation Policy Page 17 of 22

 b) EFFECTIVENESS: Extent to which an operation has achieved its objectives articulated at approval
and specified in categories such as policy goals, physical, operational, institutional, social, and
environmental, recognizing any change introduced since BoD approval.

c) EFFICIENCY: Extent to which operation’s benefits, actual or expected at time of evaluation, are
commensurate with inputs, looking at cost and implementation time.

d) SUSTAINABILITY: Likelihood that operation’s results, actual and expected at time of evaluation, will
be maintained over the intended useful project life.

e) INSTITUTIONAL DEVELOPMENT IMPACT: Extent to which the operation has contributed to

improvements or other changes in norms and practices (institutional capacities, policy framework,
externalities, etc.) that encourage cooperation and development and enable the member country(s) to
make more effective use of its human, financial, and natural resources, whether these changes were
intended under operation’s objectives or otherwise.

f) OVERALL PERFORMANCE INDEX (OPI): A single measure of overall operation’s performance

considering the evaluation findings under the 5 criteria listed under a) through (e) above.

To rate each rating criterion, several rating factors (i.e., sub-criteria) are assessed and rated. The combined
rating of the factors, associated with each criterion, comprises the rating of the criterion itself. When factor
ratings are being combined, a specific weighting is applied, to prioritise the relevance of those factors. Details
on the rating of the underlying factors that result in the rating of each criterion are maintained and updated by
the OIE.

4.4.4 Methodological Principles and Techniques

4.4.4.1 Principles

OIE deploys adequate evaluation methods, tailored to each specific case. These methods are continuously
enhanced following international best practices and harmonisation of evaluation at IFIs. The utilisation of
specific evaluation instruments varies depending on the nature, type, and functional character of the operations
and programs examined. Key milestones of the evaluation methodology include:

• Ex-post revisiting of Ex-ante. Ex-post evaluation reflects on the methods and assumptions used for
ex-ante appraisal and indicators.

• Counterfactual Approach. This is an assessment of the state of nature without the evaluated
operation/programme. The evaluation applies a "with and without the operation/program" approach or
a "before and after the operation/program" approach or a combination of both.

• Evidence-driven triangulation. This is an important principle that ensures objectiveness. The

essence is that the assessment is performed through at least three independent viewpoints/sources
that ensure checks-and-balance, with a focus on evidence. A central viewpoint is based at the OIE,
while the other two are provided through other parties involved with, or knowledgeable about, the
evaluated operation/programme. These two parties usually, but not necessarily, are (i) the Operation
Team and (ii) the Client. Other viewpoints will be considered as well, e.g., co-financier, beneficiary,
local community affected by the operation, experts, etc. The final judgement remains the sole
responsibility of OIE and is not subject to external approvals or editing – it is issued and reported by
OIE without any clearances or interference. Nevertheless, dissenting views and differences of opinion
are noted and recorded.

Independent Evaluation Policy Page 18 of 22

 • Combining Top-Down with Bottom-Up Assessments. This is another measure aiming at objectivity
by comparing the results of two different approaches: (i) the top-down approach, i.e. looking at the
targets/effects of a strategy/program and then attributing achievement to particular segments of
operations portfolio or individual operations and; (ii) the bottom-up approach, i.e. looking at the
targets/effects of individual operations and then aggregating them in order to assess if they achieved
the goals at a higher plane, e.g. at a program/strategy level.

• Attribution. It reveals to what extent the achieved impact could be attributed to different actors
involved, or not involved, with the evaluated operation/programme. These actors are usually the Bank,
the Client, the beneficiary, the government/authorities, other IFI, nature, external factors, etc.

• Quantification. To ensure objectiveness and comparability, it is important to quantify as much as

possible, though it is inevitable that many aspects will be measured through a qualitative approach.

• Results Orientation. While evaluation practice in the past has focused more on inputs and outputs,
current orientation is to focus on results, i.e., looking at outcome and impact rather than inputs-outputs
ratios.

• Lessons Learned. This is key to each evaluation as learning is central to improvement and
enhancement.

• Mission/Objective Orientation. Each evaluation makes a clear reference to the BSTDB mission,
policies, and objectives, stated in explicit and measurable terms, quantified, and standardised as much
as possible, to measure achievement vis-à-vis the benchmarks they provide.

4.4.4.2 Core Evaluation Techniques

Normally, a balanced and justified combination of quantitative and qualitative techniques is applied.
Resource/time-intensive methods need to be justified by relevance and importance of the expected outcomes.
Typically, to ensure methodological consistency, methods and techniques applied during appraisal should be
reused to the possible extent. A summary of the core evaluation techniques could be provided as follows:

• Data gathering/verification

OIE collects data by a range of methods from reconnaissance surveys to formal statistical surveys
and targeted field studies. To ensure credibility and objectiveness, each in-depth evaluation involves
a thorough on-site review. During field missions, all key sources of information are (re)visited, e.g.,
company/project site, files and management, external auditors, consultants, target groups, local
authorities, and clients/suppliers. The staff directly involved with operations support the information
gathering process by a provision in the Loan Agreement (or equivalent document) for an evaluation
visit (normally within provisions for supervision and monitoring), submission of all relevant files, data,
progress reports, briefs, contacts, etc. Upon OIE request a member of the staff directly involved with
operations will accompany the evaluator(s) during a field mission and/or will suggest a local
counterpart for such support.

• Desk Studies

Desk Studies are performed prior to and in support/complementarity of on-site reviews. They are used
to categorize and review operation/related data and progress reports. Only as an exception, Desk
Studies will be used without verification through an on-site review. This should be justified by a
marginal scale of an operation or other circumstances such as an active workout that might be
disturbed.

Independent Evaluation Policy Page 19 of 22

 • Surveys

Surveys are based on interviews and questionnaires that are used in interaction with operation
counterparts/target groups. They are normally performed along with Desk Studies and on-site reviews
to collect and process specific information. Depending on each case, a different degree of coverage
(sampling) and structuring (standardized questions and answers) will be applied.

• Cost-benefit Analysis

Cost-benefit analysis (CBA) is a form of economic analysis that normally expresses the key benefits
and costs in commensurable and actual monetary terms and arrives at a single index to determine
the value of a project. Various forms of CBA are applied, depending on the project, e.g., social CBA,
which has the perspective of an entire region/economy and considers distributional and other aspects.
CBA will normally refer to the application of such analysis at the phase of project appraisal.

• SWOT Analysis

This is another form of economic analysis that identifies Strengths, Weaknesses, Opportunities and
Threats (SWOT). An operation’s strengths and weaknesses are expressed by indicators that reveal
performance parameters. Indicators that look at opportunities and threats have a more external
nature, focusing on responses to the environment, including the production and application of policy-
related lessons. It is essential to check if a form of SWOT analysis was applied during appraisal and
implementation to verify its assumptions, using the same matrix of criteria.

4.4.5 Reconciliation of Opinions

The staff directly involved with operations may not agree with some of the findings, judgements/ratings, or
lessons learned derived by the evaluation. As the evaluation process is genuinely transparent and open to
discussion, differences will be reconciled or reflected in the evaluation report, in the form of a footnote or
attachment. However, the evaluation findings and recommendations remain a sole responsibility of OIE and
are not subject to any external approvals, consent or editing. In the event of a material pressure on the
evaluation process that might compromise its independence, OIE informs the President, the AC and the BoD,
as appropriate.

4.4.6 Cooperation with Peers

Reviewing and reflecting the practices and lessons of other Multilateral Development Banks (MDBs) enhances
evaluation and quality management. It implies close cooperation and networking with reputable evaluation units
and organizations that lead the process of coordinating/harmonizing development evaluation and provide
benchmarking of standards and best practices. The cooperation of the evaluation units of the MDBs takes place
through a dedicated international body, the ECG. OIE is a member of and regularly interacts with ECG to
implement the respective “Good Practice Standards” in due time, applying judgment on relevance to BSTDB.
While this is a continuous process, BSTDB aims to maintain full compliance with the “Good Practice Standards”.

Another form of international cooperation is to participate in joint evaluation activities in cases when common
interests, or co-financed operations. Such activities could vary from the evaluation of a co-financed project to
exchange of findings and joint missions for evaluating country/sector programs. To this end, it will be necessary
to maintain contacts with co-financiers for the purpose of coordination and streamlining of evaluation.

Adequate external and internal seminars, workshops and other forms of training will foster the transfer and
adoption of best practices and lessons in operation monitoring, evaluation, and management. Both formal and
informal contacts with other evaluation units will be translated into a tailored transfer of valuable know-how, as
well as exchange/secondment of evaluation professionals. Interactive and participatory training of existing and

Independent Evaluation Policy Page 20 of 22

new BSTDB staff members should reveal useful lessons and practices from other institutions to avoid
duplication of efforts or repetition of failures.

5 References

• “Good Practice Standards” regularly updated by the Evaluation Cooperation Group of the Multilateral
Development Banks

• 52nd AC meeting minutes on Independence of Evaluation and ECG membership

• BoD decision BD-13-020-93-802013-049, related to the above

• Operations Cycle Policy

• Operations Manual

• Public Information Policy

6 Associated Documents

6.1 List of Records & Forms

The table below lists all records to be produced in the course of the application of the Policy

1 Annual Evaluation Overview Reports

2 Operation Performance Evaluation Reports (complete or summary)

3 Evaluation Studies

4 Operation Completion Reports

5 Lessons Learned Memoranda

6 Mid-Term Evaluation Reports

7 OIE Budget

8 OIE Work Program

Independent Evaluation Policy Page 21 of 22

7 Distribution List

This Policy is accessible by all employees of the Black Sea Trade & Development Bank. OIE performs
clarifications and trainings regarding the implementation and updates of the Policy.

The following entities should be informed about the approval of this Policy:

• Senior management and all staff

Independent Evaluation Policy Page 22 of 22