Breese-MeasuresAndBenefitsRealization

Exhibit E
2017 RESEARCH INFORMED STANDARDS
Measures for benefits realization
TABLE OF CONTENTS
1. Executive Summary Page 2
2. Introduction, questions asked and summary answers Page 6
3. Methodology Page 10
4. Findings – Detailed responses to each research question Page 14
5. Reflections Page 69
6. Conclusion and Recommendations Page 76
7. Appendices Page 79
NOTE: APPENDICES include all papers considered in the preparation of this

project
1
1. Executive Summary
This analytical literature review on ‘Measures for benefits realisation’ is concerned with metrics
and the measurement of outcomes in benefits realization. The brief was to address six
questions, as outlined below.
1. What is the ‘state of the art’ in measuring benefits?
2. At what point(s) in the project are outcome benefits measures developed, defined and
selected?
3. Who assesses the benefits and at what point during the project are they assessed?
a. Are measures added over the life of the project and/or beyond?
b. How far after the close-out of the project are benefits continued to be assessed,
and at what intervals?
4. Does this vary by project type (e.g., change project, innovation or new product
development, etc.) or by industry, project size, potential social impact, or even who the
customer is?
5. What kinds of measures are typically used to assess benefits, specifically Quantitative
and/or Qualitative, and which are more frequently used? Does this vary by the same
dimensions as #4 above?
6. What happens with a gap between benefits accountability and project implementation?
The research undertaken was a literature review of documentation in the English language.
Four different sets of literature have been investigated
- Government body (aimed for comprehensive coverage of documents –67 in total)

- Professional body (aimed for comprehensive coverage of documents – 42 in total)
- Consultant/practitioner (systematic coverage limited to books and published consultant
reports – 24 in total).
- Academic (literature search identified 127 relevant academic sources).
Having analysed these 260 documents against the six questions, certain headline characteristics
of the current position on benefits measurement emerged. These led the team to draw
conclusions and make eight recommendations. These are outlined below.
2
The study has identified a ‘knowing-doing’ gap in benefits measurement, whereby there is a
gap between what should be and what is, in terms of ‘state of the art’ as described in the
guidance, and the ‘state of the art’ as evidenced in current practice. Guidance is accessible but
there is less evidence easily available describing existing practice, because much of that
evidence is in academic sources. Therefore,
Recommendation 1 is that there should be more initiatives to make the evidence on practice
on benefits measurement and management more accessible to practitioners and other
interested parties.
There are terminological inconsistencies affecting benefits measurement, for example, in the
different relationships between the terms ‘outcome’ and ‘benefit’ in different guidance
sources. There are also a variety of different dualities (paired terms) used to categorise
outcomes and/or benefits, such as intermediate/end, leading/lagging and short term/long
term. The roles of projects, programs and portfolios in benefits realisation translate across into
benefits measurement, so clear terminology is required to enable clear and consistent
messages to be provided in guidance. This will help with clarity on what needs to be measured
and how it is to be measured (in conjunction with Recommendations 3 and 5 below, on
ownership of the process). Therefore,
Recommendation 2 is that consistent and clear sets of terms be developed for guidance on
the causal relationships involved in strategic alignment of project activity, incorporating
program and portfolio levels where organisations utilise these levels, which will in turn
facilitate the benefits measurement process. This links especially directly into the use of
benefits dependencies/mapping tools.
P3M does not exist in isolation - it exists within the broader context of overall enterprise
governance and management - from strategy through to operations. The diverse roles of
different stakeholders in all aspects of P3M are increasingly recognised, but the stakeholder-
specific implications of benefits targets are not always appreciated. There is a need for clear
ownership of the BRM process and for the owners to involve key stakeholders in identifying
benefits and developing benefits measures and targets, and then also keep them engaged
during the benefits life-cycle, so the whole organisation is committed to the optimisation of
benefits. Key stakeholder groups include users and may include funders, but could also cover
any individual or group with high power and high interest. Therefore,
Recommendation 3 is that guidance needs to build on progress towards owners of the BRM
process engaging key stakeholders in the development of benefits measures and
subsequently throughout the benefits life-cycle. Useful tools include RACI frameworks,
stakeholder workshops and other techniques to link benefits to organisational priorities.
3
Developing the theme of the benefits life-cycle, there has been some progress towards the
incorporation of an emergent approach to benefits measures and an emphasis on the post-
implementation phase as a focus for benefits realisation. However, this needs to go further.
Therefore,
Recommendation 4 is that guidance needs to emphasise the importance of benefits

measurement and management over the whole life-cycle, taking an emergent approach that
stresses benefits realisation post-implementation.
While there are many areas in which progress is being made, a key challenge remains
accountability for benefit identification and realisation at the Executive and Board levels in
organisations, which is essential if potential long term benefits after handover are to be
achieved. Equally well, operational managers can compromise the achievement of long term
benefits if they do not take ownership for the benefits and therefore do not see benefit
realisation as an integral part of their role, linked to strategic priorities. Therefore,
Recommendation 5 is that the P3M community needs to explore further ways of encouraging
an enterprise-wide culture of value, from the Board through to operations, to enable
potential benefits from investment in change to be realised.
Identification of benefits measures, addressing measurement issues such as quantification and

monetisation, setting targets, tracking benefits, incorporating emergent benefits and
maximizing long term commitment to the measurement and management of benefits all have
many challenges. Some of these are technical, but they mainly concern behaviours and
attitudes. Generally these issues have to be addressed in relation to the opportunities and
constraints in different organisational contexts. Therefore,
Recommendation 6 is that guidance needs to set a framework which enables different

categories of investment and business sectors to adopt approaches to benefits measurement
and management which fit their own context.
The study has found that there is much descriptive evidence on benefits measurement
available, but that there are also many research gaps, some of which might be filled through
inter-disciplinary working with researchers in areas such as change management, investment
appraisal, evaluation and performance measurement/performance management. However,
there is also a need for further research into many areas of benefits measurement and
management. Examples include
- In-depth studies of the whole benefits life-cycle

- Comparative research across different dimensions, e.g. different industrial sectors and
public/private/third sector
4
- Roles in benefits measurement and management, particularly in terms of the variety of
individuals and groups involved in different tasks at different levels
- The use of different benefits dependencies/mapping techniques and the influence of
terminology on their effectiveness.
Therefore,
Recommendation 7 is that opportunities to increase inter-disciplinary collaboration with

allied research areas should be promoted.
and
Recommendation 8 is that efforts to address priority research gaps in the field of benefits
measurement and management be made.
5
2. Introduction, questions asked and summary answers
This analytical literature review is concerned with metrics and the measurement of outcomes in
benefits realization. Our brief was to address six questions, as outlined below, with details of
the proposed approach:
1. What is the ‘state of the art’ in measuring benefits?
Under this question, both normative guidance on how to measure benefits and
descriptive evidence on existing practice will be included. We will look across the broad
spectrum of approaches to develop a speculative analysis (best “guestimate”) of where
the field is headed with regard to measuring benefits as well as a perspective on which,
of the emerging measures, we anticipate may prove most useful.
The study will cover project, programme and portfolio levels. In the questions below,
wherever ‘project’ is mentioned, our study will also incorporate programme and
portfolio levels.
2. At what point(s) in the project are outcome benefits measures developed, defined and
selected?
This question is concerned with timing, but will also address different types and levels of
benefit. Particular attention will be paid to measures for ‘outcome’ benefits’ (other
terms are sometimes used, such as ‘final’ or ‘end’ benefits) but the measures for other
‘intermediate’ benefits will also be analysed, and interdependencies between
intermediate and outcome benefits investigated.
We will seek to identify literature that maps different stakeholders to stakeholder-

specific benefits (intermediate or outcome).
3. Who assesses the benefits and at what point during the project are they assessed?
The term 'assess' in the question is taken to include identification, monitoring,

management and review. We will seek to identify who is involved in determining what
benefits will be measured and how they will be measured. ‘Who’ will include individual,
project team, organization-level and wider stakeholder responsibilities. The prevalence
of unplanned/emergent benefits will be investigated.
6
4. Does this vary by project type (e.g., change project, innovation or new product
development, etc.) or by industry, project size, potential social impact, or even who
the customer is?
The dimensions eventually chosen for this question will depend on the amount of
evidence available. For example, public/private/voluntary sector may be a key
categorisation, and perhaps degrees of project risk. Various different ways of
categorising projects will be investigated by grouping industries according to metrics-
related characteristics. For example, application development projects vary from
physical construction projects in the relevance of interim measures.
5. What kinds of measures are typically used to assess benefits, specifically Quantitative
Under this question we will investigate issues such as: whether the type of measures has
changed over time? Are mixed methods used? Are there specific
industries/environments/project types that rely on one kind of measure more than
another? If so, what is/are the underlying reasons?
We will include the use of methods of ‘quantifying’ qualitative measures, such as Likert
Scales for attitudes and feelings, in relation to the balance of quantitative/qualitative
metrics.
6. What happens with a gap between benefits accountability and project

implementation?
This question will be concerned with the focus on BRM being lost during
implementation of the project or after it has been completed, specifically in relation to
accountability for benefits measurement. It will link closely to Question 2 (timescale
issues with metrics) and Question 3 (‘who’ is responsible for metrics) since the
development of a gap between benefits accountability and project
implementation represents a particular scenario in terms of metrics, reflecting a
downgrading over time in the priority given to benefits measures by the people
who should be responsible for the benefits life-cycle.
7
The research project was carried out in the second half of 2017, based on the schedule of
activities outlined below.
15-May 15-Jun 16-Jul 16-Aug 16-Sep 17-Oct 17-Nov 18-Dec
Review, clarify and finalise scope

Project Initiation
Update literature source lists
Develop templates for literature analysis
Detailed plan submitted
Government Body literature - analysis
Professional Body literature - analysis
Practitioner/Consultant literature - analysis Duration
Academic sources literature - analysis
Guidance on metrics in BRM - results
Interim summary of findings - normative
Update source lists (June - October, 2017)
Existing practice in metrics for BRM - results
Review of other literature domains
Synthesis of findings/prepare final report
Submit final report
The research undertaken was a literature review of documentation in the English language. As with the
team’s first report for the PMI, four different sets of literature have been investigated
- Government body (aimed for comprehensive coverage of documents –67 in total)

- Professional body (aimed for comprehensive coverage of documents – 42 in total)
- Consultant/practitioner (systematic coverage limited to books and published consultant reports
– 24 in total).
- Academic (literature search identified 127 relevant academic sources)
The next section is ‘methodology’, which outlines how the framework for undertaking the literature
searches was decided upon, and the detailed methods used for each review and the presentation of the
results.
The ‘findings’ section addresses the six questions, summarizing the relevant information from each
literature type.
The ‘reflections’ section is concerned with the main implications of the research. It sets out the main
conclusions for each of the six questions.
The ‘conclusions and recommendations’ section makes eight recommendations based upon the analysis
carried out in the report. These recommendations are concerned with the promotion of good practice in
8
benefits measurement and management. We found no shortage of guidance on how to undertake
benefits measurement, but the descriptive evidence of benefits measurement practice does not match
up to it. There is also a shortage of detailed case study material, with the exception of the IT-enabled
change field. Our report draws attention to guidance and examples which could be used to illustrate
how to undertake benefits measurement, but our conclusion from this evidence and our collective
experience is that there is a gap between what should be and what is, in terms of ‘state of the art’
as described in the guidance, and the ‘state of the art’ as evidenced in current practice.
Therefore, the report recommendations concentrate on a variety of actions of different types
which might help to close that gap.
The report has been prepared by
Dr. Richard Breese, DBA, MAPM, FHEA (Sheffield Business School, Sheffield Hallam University) (Principal
Investigator)
Stephen Jenner, CIMA, MBA, FCMA, FAPM (Portfolio Solutions)
Carlos Serra, PMP, PRINCE 2, MCT (Independent researcher)
John Thorp, CMC, ISP, ITCP (The Thorp Network)
Dr. Amgad Badewi, PhD PMP, MSP AP, ITIL (Kent Business School, University of Kent)
Michael Charlton, BA, MBA, FHEA (Sheffield Business School)
9
3. Methodology
The original brief for the work specified that the report cover projects and, if possible, programs and
portfolios as well. A key principle shared by the team is that where organisations utilise programs and
portfolios as well as projects, benefits management/benefits realization management (BM/BRM) has to
be applied across all three levels, so this was our starting point.
The method built on that employed for our first BRM RIS project, adapted to fit with the nature of the
brief for ‘Measures for benefits realization’. The analytical literature review was based upon the four
literature areas in the first study. Since the first project was undertaken, further documents have been
published, such as the PMI’s own Thought Leadership Series on Benefits Realization Management.
Therefore, one of the first tasks was to update the literature searches for each of the four areas, with
the focus being primarily on the topic of ‘measures’ and the context in which BRM uses them, with
additional literature specific to this topic.
Following this exercise, the number of sources covered was,
- Government body 67 sources (58 in the first study)

- Professional body 42 sources (31 in the first study)
- Consultant/practitioner 24 sources (21 in the first study)
- Academic 127 sources (97 in the first study)
A further preliminary task was to develop the templates to be used for each of the four literature areas.
It was decided to draw information to address the final question, Question 6, from the analysis of the
other five questions. Even so, the five questions and the sub-elements within them meant that it was
difficult to fit all the data to be collected into tabular format, without the columns becoming narrow. We
therefore experimented with different approaches, with the result that the format of the tables for the
Government Body and Consultant/practitioner reviews are slightly different from those used for the
Professional Body and Academic reviews. The column headings nevertheless ensured that data was
collected systematically and consistently across the different sources in each literature area, and
ensured that the information recorded mapped onto the research questions.
The investigation of measures has two main areas, which cut across all six questions,
- guidance of a normative, ‘how to do it’ nature

- descriptions of existing practice, which sub-divide further into, for example, best practice and
typical practices.
One of the final tasks was to review the evidence that is available from the literature and identify gaps,
as they affect the six questions. For example, under the ‘investment type’ dimension in Question 4,
10
there is a lot written about IT-enabled change, but little on other sectors. A brief section in the
reflections chapter is therefore devoted to other literature domains which might help to fill some of
those gaps, from a different perspective (perhaps addressing benefits measures issues, but using
different terminology).
The following sub-sections outline the sources used and search methods for each literature review.
Government Bodies Literature Search
We aimed to be comprehensive for sources in the English language. Publications from the following
jurisdictions/bodies were reviewed, including some new sources recently published or ones which were
particularly relevant to the subject matter for the research project,
 United Kingdom: OGC (Now Axelos – a joint venture between the Cabinet Office and Capita); HM
Treasury; Cabinet Office; and the National Audit Office.
 UK Devolved administrations: Welsh Government & N Ireland Department of Finance & Personnel.
 Europe – Germany.
 Canada Treasury Board Secretariat.
 USA - Office of Management and Budget (OMB); GAO; Legislation (Program Management
Improvement and Accountability Act 2015; Government Performance and Results Modernization
Act 2010; and Government Performance and Results Act 1993); White House Circular No. A-94
Revised; CIO Council; National Electronic Commerce Coordinating Council; and Intergovernmental
Advisory Board, Federation of Government information Processing Councils.
 Australia – federal government and state governments – NSW, Queensland, Tasmania, Western
Australia, and Victoria.
 New Zealand
 Inter-jurisdictional guidance including: the Better Business Cases initiative (UK, Welsh and New
Zealand governments); World Bank; OECD; and European Commission-funded studies.
67 sources were reviewed, compared to 58 in the first study.
In the relevant section in Chapter 4 (Section 4.1), government bodies sources are cited using
‘GSR’ for ‘government sources review’, with a number which corresponds to the first column in
Appendix 1.
Professional Bodies Literature Search
Publications from the following bodies were reviewed: Project Management Institute (PMI);
Association for Project Management (APM); APMG-International; Australian Institute for
Project Management (AIPM); International Project Management Association (IPMA); Change
Management Institute (CMI); British Computer Society ( BCS); ISACA (Previously known as the
11
Information Systems Audit and Control Association); International Centre for Complex Project
Management (ICCPM); and relevant ISO standards.
The analysis of the publications from these professional bodies is contained in Appendix 2,
listed in order of professional body.
The analysis includes some new sources which have been published in 2016 and 2017, including
the PMBoK (6th edition), a series of Thought Leadership reports on Benefits Realization
published by the PMI and new publications from the APM, ISACA and ISO. 42 sources are
covered, compared to 31 in our 2016 report to the PMI on BM/BRM terminology.
In the relevant section in Chapter 4 (Section 4.2), professional bodies sources are cited using
‘PBR’ for ‘professional bodies review, with a number which corresponds to the first column in
Appendix 2.
Consultants/Practitioner Literature Review
The systematic literature review covered
- Practitioner/consultant books
- Consultancy publications.
As with the first study, other categories of consultant/practitioner outputs were not used, because of
the reduced credibility of such sources compared to the ones we did use.
24 sources were reviewed, compared to 21 in our first study.
In the relevant section in Chapter 4 (Section 4.3), consultant/practitioner sources are cited
using ‘CPR’ for ‘consultants/practitioners review’, with a number which corresponds to the first
column in Appendix 3.
Academic publications Literature Search
The Academic review for our first report for the PMI, on terminology for Benefits Realization
Management, used search terms in order to identify a long list of publications, which was then
narrowed down to a manageable number of the most relevant documents.
12
The search terms used were:
 Benefits+Management
 Benefits+Realis(z)ation+Management
 Benefit+Management
 Benefit+Realis(z)ation+Management
These terms were fed into three databases
 Google Scholar(title only)

 Science Direct (keyword)
 EBESCO (Business Source Premier) (keyword)
After the results were manually screened, this resulted in a list of 97 academic publications.
For this second study, a search was undertaken using the same method for additional
publications from 2016 and 2017, which post-date the searches for the first study. In addition,
further books and academic theses which did not come up on the search results, and older
sources referenced in the literature of particular relevance to benefits measures were added.
This resulted in the total number of academic sources rising from 97 to 127.
The academic literature review was developed using a Word table (Excel was used for our first
report), with separate documents for the original 97 publications (Appendix 4) and for the 30
additional ones (Appendix 4a). In the relevant section in Chapter 4 (Section 4.4), sources from
the original list are cited using ‘AOR’ for ‘academic original review’ publications, with a number
which corresponds to the first column in Appendix 4. Sources from the new list are cited using
‘ANR’ for ‘academic new review’, with a number which corresponds to the first column in
Appendix 4a.
13
4. Findings – Detailed responses to each research question
Introduction
The findings are structured based upon each literature review, rather than using the questions as the
primary structuring dimension. We took this approach because of the overlap between the questions. It
seemed that following through the characteristics of a group of sources against the questions in turn
would help the flow of the argument in this particular context.
The reflections chapter is the place for a summary of the key points against each question to be covered,
based upon the combined results of the four literature areas.
In reviewing the scope of the four literature areas, in general, the boundaries were clear. The main grey
area concerns books on BRM. If books are written by academics, and draw from material which has also
appeared in academic journals or academic conferences, then they are included in the academic
literature review, e.g. Ward and Daniel (AOR88). If the book has been published by a Government
agency or professional body and appears to represent the policies/views of that organisation, it is
included in the Government Body or Professional Body review, e.g.Bradley, ‘Fundamentals of Benefits
Realisation’ (OGC) (GSR6). Other books were included in the Consultants/Practitioner literature review.
It is also the case that sources relating to a specialist subject matter might be split between the four
reviews. For example, in this study new sources relating to Social Return on Investment (SROI) have
been included, because of the relevance to BM/BRM. It happens that the three publications discovered
fit in three different literature reviews. However, we have cross-referred across the boundaries of the
literature areas where this is the case.
14
4.1 Government Body Sources
Q1: What is the “state of the art” in measuring benefits? (Normative/descriptive, Project,
Program or Portfolio)
Four broad categories of guidance were reviewed
 BRM guidance issued to/by the P3M community.

 Cost-benefit/business case guidance issued by central finance bodies.
 e-Gov benefits literature.
 Literature from audit bodies on how well the guidance is working i.e. to what extent is
the guidance applied and does it work?
The sources in each category are:
 Government PPM Benefits Management guidance – OGC/Axelos MSP® (GSR1) guidance

in particular. Key sources include – UK (MSP (GSR2) & Bradley (GSR6), MoP® (GSR3),
Northern Ireland), Canada, New Zealand, and Australian State governments, particularly
Victoria and NSW.
 Business Case & Cost-Benefit Analysis Guidance - Key Sources: HMT Green Book
(GSR20), BBC initiative (Wales, England, NZ) (GSR32), SROI (GSR27), US Circular No. A-94
Revised (GSR40), and Queensland (GSR53, GSR55).
 e-gov benefits - Key Sources: HM Treasury (GSR25), VMM (GSR47), DVAM/NOIE study
(GSR48); Intergovernmental Advisory Board, Federation of Government information
Processing Councils (GSR36); OECD (GSR45); Foley (GSR64); and eGovernment
Economics Project – Measurement Framework Final Version (2006, GSR65).
 Literature on measurement/forecasting accuracy and benefits realisation measurement
in practice – Key Sources: Mott McDonald report (GSR22), Green Book supplementary
guidance (GSR23), and the NAO (GSR28, GSR30, GSR31).
A key finding is that there is a great deal of consistency between countries/jurisdictions within
each of the above categories – but significant differences in focus and approach between the
categories.
The most significant category for the study is the first one, BRM guidance issued to/by the P3M
community. There is a high degree of common ground between the sources, often reflecting
the OGC/Axelos MSP® (GSR1, GSR2) guidance in particular. Key sources include – UK (MSP
(GSR2) & Bradley (GSR6), MoP®(GSR3), Northern Ireland (GSR34)), Canada (GSR45), New
Zealand (GSR59), and Australian State governments, particularly Victoria (GSR58) and NSW
(GSR51). These sources focus on generic approaches to benefits measurement (who does it,
when, using which documents etc) with less focus on methods to measure benefits (which is
15
the focus of the cost-benefit/business case guidance – at least for monetisable benefits; and
some of the e-gov literature). It generally has less of an emphasis on what specific measures
can be used for specific benefit categories (which is the focus of the e-gov literature, and the
HMT Green Book (GSR20) also includes guidance on valuing a range of social and environmental
benefits). However, the New Zealand Government has prepared a Social Outcomes Catalogue
(GSR Additional Note).
Most of the Government Body sources are exclusively concerned with normative guidance,
although there are examples where descriptive information is provided. The breakdown is,
 Normative: 47 Sources
 Normative with examples/case studies or data on benefits measurement: 8 Sources
 Descriptive: 12 Sources
The sources include some orientated specifically to projects, programs or portfolios, but a
majority of sources were orientated towards projects and programs, with some publications
concerned with the integration between all three levels. The breakdown is,
 Project, Program and Portfolio – 10 Sources

 Project & Program – 32 Sources
 Portfolio – 2 Sources
 Program – 9 Sources
 Project (including e-Gov initiatives) – 14 Sources
Q2: At what point(s) in the project are outcome benefits measures developed, defined
and selected?
24 sources did not address the question. Of the 43 that did, all sources agreed that pre-
investment measurement is required. Common features identified were:
 The Benefits Management Strategy describes how benefits will be measured and by
who on a program/project.
 Specific measure(s) for each benefit should be identified - at least one per benefit,
although some sources emphasise using several measures to obtain a more-
informed view on benefits realisation.
 Existing organisational measures should be used where possible and be linked to the
performance management system; and a standard set of measures should be
developed linked to strategic objectives/KPIs for use by all projects and programs to
demonstrate strategic contribution.
16
 Benefit measures identified are recorded on a Benefit Profile/Register prepared
alongside the Business Case, with the forecast scale of impact (some sources refer to
targets) also being recorded in the Benefits Realisation Plan.
 Business Cases do not always clearly identify the benefits and when they do they are
commonly over-stated due to optimism bias and strategic misrepresentation.
Solutions identified include: optimism bias adjustments and reference class
forecasting; sensitivity analysis; confidence ratings; challenge; and stronger
accountability.
In the Business Case & Cost-Benefit Analysis guidance there is
 A focus on benefits being measured primarily as part of project appraisal (forecasts) and
at the evaluation/post-implementation stage – although the emphasis is on the former.
 Emphasis on measuring both financial and nonfinancial benefits in financial terms for
inclusion in a NPV calculation in the Business Case.
 Measures are based on market rates for financial benefits. Where market prices are not
available, techniques such as stated and revealed preference can be used to value
nonfinancial benefits.
Only 14 sources out of the 57 sources distinguished between outcomes/end benefits and
intermediate benefits, using dualities (paired terms) such as intermediate/end,
intermediate/ultimate leading/lagging and short term/long term. Examples are
 N. Ireland (GSR34) distinguishes between: Intermediate benefits, “these describe the

actual operational improvement resulting from the programme or project, for
example quicker access to information, improved financial management, faster
turnaround times; these functional or operational benefits must be able to be
measured and if an intermediate benefit can’t be measured then it isn’t a useful
benefit - an individual benefit profile is produced for each intermediate benefit and
this is used to record the benefit, allocate responsibility for measuring it and to
identify any activities required to manage benefit delivery” & End benefits, “these
are strategic or organisational level benefits or benefits linked to the wider NICS and
are generally aligned with organisational strategy and corporate plans; end benefits
usually describe what the organisation is seeking to achieve as a result of the
business changes and measurement is achieved through the measurement of its
component intermediate benefits”.
 Bradley (GSR6) and OGC 2005 (GSR12) specifically refer to intermediate and end
benefits and measuring each; SROI (GSR27)refers to monitoring intermediate
benefits to provide a measure of ‘distance travelled’.
 MSP (GSR2) Refers to short and longer-term benefits in a causal chain (shown on the
benefits map).
17
 Canada Outcome Management (GSR45) distinguishes between intermediate and
ultimate outcomes: “identification of intermediate outcomes that serve as
milestones or leading indicators towards attaining the outcomes and to permit
tracking of progress towards the final outcomes” & “To achieve the end results of an
initiative, it is crucial to identify and track intermediate outcomes that can be used as
milestones along the road.”
 OGC (undated, GSR10) and Cabinet Office, 2017 (GSR16) refer to leading and lagging
measures.
It should be noted that end benefits depend on intermediate benefits, but the reverse is not
the case.
Many Benefits Management Guidance sources emphasise the importance of agreeing benefit
measures with relevant stakeholders e.g. via facilitated workshops. Some sources from
different categories of guidance link benefits/measures to stakeholder groups:
 Better Business Cases (GSR33) guidance: “Public Sector benefits – those falling to the
spending organisation, over which it has direct control of their realisation (Direct Benefits)
and those falling to other parts of the public sector (Indirect Benefits); Wider Social benefits
– those other indirect benefits falling to other sectors, including the private sector.”
 IPA 2017 (GSR19): government, private sector partners and wider UK public.
 HMT eGOV guidance 2003 (GSR25): Customer - Business & Citizens, Government, & Society.
 US VMM (GSR47) – 5 sources of value: Direct User (Customer) value; Social (non-direct
user/public) value; Government operational/foundational value; Government financial
value; Strategic/political value; and by type: Govt to Citizen; Govt to Govt; Govt to Business;
and Internal Efficiency and Effectiveness.
 Australian DVAM (GSR48) - encompassed five forms of value: Agency benefits/value;
Strategic value; Consumer financial benefits; Social benefits; and Governance value with
three indicators: increased community participation in democratic processes; increased
transparency of government processes; and increased accountability.
 OECD 2006 (GSR45) Benefits were categorised into the following categories: Benefits to
Government (37 indicators were identified under the headings: Direct Cash benefits,
Efficiency savings (monetisable benefits) and other non-monetisable benefits); Benefits to
Citizens/Users - 29 indicators were identified under the headings: Monetary benefits; Non
Monetary Time-based; & Non Monetary Value-based.
18
Qu 3: Who assesses the benefits and at what point during the project are they assessed?
Who measures the benefits is not addressed/specified by 48 Sources (many address

measurement but don’t specify who should do the measuring). Of the 19 sources that do
address this issue, most specify that it is a business responsibility falling to:
 Business Change Manager – MSP (GSR2), OGC 2006 (GSR12)

 Benefit Owner – NSW BRM Framework (GSR52) & New Zealand: the benefit owner
“Collects and reports data to evidence benefits realisation”
 Bradley (GSR6) refers to “Measure monitors” with this role being shared among the
stakeholders; as well as, “Benefit owners, who may be tracking the benefits
themselves”
 N. Ireland (GSR34) guidance distinguishes between an “operational benefit owner –
a business representative who is responsible for benefit measurement and the
management of any activities required to ensure benefits are realised” and “A senior
benefit owner is a senior business representative responsible for ensuring that the
benefit is achieved once handover from the programme or project is complete…the
actual measurement, monitoring and management of benefits will be done by
named operational benefit owners and overseen by senior benefit owners.”
 Canada Outcome Management (GSR45) - Outcome Owner’s who are, “responsible
and accountable for achieving his or her target. In addition, there is often an
additional accountability for reporting on the metric, to distinguish between the two
activities and responsibilities.”
 Victorian ILM (GSR55) refers to a Benefit data provider: “A person who has been
identified as the custodian of data that will be required as evidence that a KPI has
been met.”
Other sources highlights the facilitating/coordinating role of PMO staff:
 Portfolio Benefits Manager/Benefits Realization Manager role – in setting the rules

on quantification and valuation of benefits and consolidating progress reports for
the portfolio dashboard report and maintaining the portfolio-level benefits forecast
(MoP (GSR3), Venning (GSR11)).
19
 P3O (GSR4) refers to a Benefits role in the Project/Program/Portfolio Office who
collates and facilitates measurement by business managers.
 MSP (GSR2) refers to a similar role being performed by the Program Office.
Many sources emphasise the importance of stakeholder engagement in benefits measurement

– for example:
 SROI (GSR27) places “stakeholders at the heart of the measurement process” (NEF)
 VMM (GSR47) “Valid results depend on project staff working directly with
representatives of user communities and partner agencies to define and array the
measures in order of importance”
 Canada Outcome Management (GSR45)– measures should be agreed with the
Outcome Owners
 DVAM (GSR48) - Emphasis on user involvement in identifying benefits, rating/scoring
them and assessment of performance
 Victorian ILM (GSR55) - Measures are developed with business stakeholders - “A
benefit definition workshop is used to identify the KPIs, measures and targets that
must be met to mitigate the effect of the problem.”
 World Bank (GSR67), “Participation of key stakeholders in defining indicators is
important because they are then more likely to understand and use indicators for
management decision-making.”
 Tasmania (GSR57) – measures should be agreed with the Benefit Owners.
There were two main groups on the approach to assessment over the benefits life-cycle. These
were
 Continuous or Regular/Periodic measurement: 3PM Guidance that emphasises

continuous or at least regular measurement throughout the business change lifecycle
e.g. Bradley (GSR6), MSP (GSR2), MoP (GSR3), New Zealand, Canada (GSR45), N Ireland
(GSR34) etc. From baselining (e.g. OGC (GSR1), 2006, “in particular ensuring that the
‘before’ state is measured so that an assessment can be made as to whether the ‘after’
measurements indicate an improvement or not”), then tracking post-implementation at
end of tranche and other benefit reviews, through to (and in some cases beyond) post-
implementation review. For example:
o Australian Fed Govt (GSR46) (2012) - Benefits realisation is maximised where:
“Benefits are measured routinely and are part of normal planning and reporting
functions—not regarded as an optional and stand alone exercise”; and
o N. Ireland (GSR34) – “A pre-implementation baseline measurement followed by
defined actual measurements at relevant points during, and post,
implementation.”
20
 Pre and post investment measurement: Cost-Benefit Analysis Guidance (such as HMT
Green Book (GSR20), Optimism Bias Guidance (GSR21), Better Business Case Guidance
(GSR33), White House Circular No. A-94 Revised (GSR40), NSW Treasury (GSR51) etc.)
focuses on measurement for project appraisal and to a lesser extent, evaluation (at the
finish). Other sources with this focus include: SROI (GSR27) and e-Gov frameworks such
as the US CIO VMM (GSR47) and Australian DVAM (GSR48). The focus is therefore on
forecasting for the appraisal stage and then considering whether the forecast benefits
have been realised – as expressed by the NSW guidance (GSR51) - “Benefits realisation
is an established practice of ensuring that projects or programs produce the anticipated
benefits claimed in the project’s economic appraisal”.
Are measures added over the project? Emergent benefits?
This is was not addressed by 52 sources. The 15 sources that include identifying & measuring
emergent/unplanned benefits, include
 NSW (GSR52) BRM Framework - “The process of organising and managing so that
potential benefits arising from investment in change, are actually achieved.”
 MSP (GSR2) – a Business Change Manager role.
 P3O (GSR4) – benefits role to assist business managers/BCMs, “to identify additional
opportunities for benefits realization”
 SROI (GSR27): unintended benefits should be identified at the evaluation stage.
 N. Ireland (GSR34) - review for emergent benefits at summary review points.
 New Zealand (GSR59) focus on unplanned/emergent benefits across the BM
lifecycle.
It should be noted that the Cost-benefit and Business Case guidance focuses on planned
benefits with little reference to emergent benefits.
Measurement post-project & when?
This was not addressed by 55 Sources. Regarding the 12 sources that did cover it, the following
points are noteworthy.
 MSP (GSR2) - measurement continues after the program by BAU/operational

managers and can be the BCM. New Zealand guidance (GSR59) also sees the BCM as
being accountable for benefits measurement following program closure.
 MoP (GSR3) - One of the 6 main elements of the Portfolio-level Benefits
Management Framework is, ‘Effective arrangements to manage benefits post
project / programme closure’.
21
 Bradley (GSR6) sees post-closure measurement falling to the Benefit Owners –
“Their role will continue beyond the life of the programme”. Similarly, in the N.
Ireland guidance responsibility lies with the operational benefit owner; and the DWP
Case Study (NAO, 2006 (GSR28, GSR30, GSR31)) refers to a benefits realisation plan
that assigns responsibility for securing benefits to named individuals – benefits
owners – over the medium-term of the first few years’ following implementation.
 Canadian Outcome Management (GSR45)– “The outcomes monitoring and reporting
process continues until all outcomes are realized and stable, particularly if the
benefit pertains to a measurable performance level.”
 Aus Fed Govt 2012 (GSR46) – the project/program must ensure, “the responsibilities
for measurement are transferred to an appropriate (Agency) corporate area as part
of its project/program closure activities.” Queensland Project Assurance Framework
– 2 (GSR53, GSR55). Benefits Realisation, “a succession plan to handover any
benefits management plans, supporting benefit profiles and reporting responsibilities
to the appropriate business owner”. NSW BRM Framework (GSR52) – “In order for
benefits to be tracked after a program has ended there needs to be clear
identification of the owners of benefits within the business, and effective handover of
benefits measurement and reporting to the business owner.”
Qu. 4: Does this vary by project type (e.g., change project, innovation or new product
the customer is?
The guidance is generic across project type, industry and size, except for those directed towards
e-Gov. Initiatives. In terms of who the guidance is directed to, this breaks down into
 3PM practitioners & managers, Cross-sector, global scope – 15 Sources

 Central/State/Federal government specifically SROs, 3PM staff, business case writers – 38
Sources
 e-Gov initiative participants/policy makers – 13 Sources
 3rd sector initiatives – 1 Source (SROI (GSR27), where potential social impact is emphasised).
The Government Body literature therefore provides little help in identifying variations in
benefits measurement practice across different dimensions.
Q5: What kinds of measures are typically used to assess benefits, specifically Quantitative
22
36 sources do not distinguish between quantitative and qualitative benefits, with most of such
sources focusing on quantitative measures. 31 sources do distinguish between quantitative
and qualitative measures (or use similar/equivalent terms).
Sources referring to qualitative benefits/measures include those that see qualitative as of less
value than quantitative measures - for example, HMT Green Book (GSR20), IPA 2017 (GSR19)
and Better Business Cases Guidance - “ascertain whether the benefits are economic (non-cash
releasing) or financial (cash releasing); measurable, but not in cash terms; or simply
qualitative.”; “cash releasing benefits (CRB). non-cash-releasing benefits (non-CRB). quantifiable
benefits (QB). non-quantifiable benefits (non-QB). These are the qualitative benefits, which are
of value that cannot be quantified.” & “Qualitative benefits (Qual) cannot be measured nor
monetized (meaningfully).”
Those that recommend combining quantitative and qualitative measures include:
 New Zealand (GSR59): “Qualitative (intangible) categorisation Benefits expressed in

descriptive terms e.g., satisfaction rating. Quantifiable (tangible) categorisation
Measures expressed in numerical terms e.g., hours saved, income generated etc.”
 SROI (GSR27) - A mix of subjective or self-reported (e.g. what the stakeholder says)
and objective indicators (data) are recommended;
 N. Ireland (GSR34): “Quantitative benefits can easily be measured numerically;
qualitative benefits are normally measured with a questionnaire and the response is
measured numerically.” NI distinguishes between: Direct monetary benefits
(tangible); Direct non-monetary benefits (tangible); and Indirect benefits
(intangible) - can be identified, but cannot be easily quantified for example end user
satisfaction, better access to information, organisational image, customer service,
better morale, better perceptions.
 Canada Outcome Management (GSR45)- Outcomes are classified as: Direct
Quantitative – labour savings, Direct Quantitative – other direct savings, Indirect,
and Qualitative; “Adopting both quantitative and qualitative measures for outcomes
presents a broader view of expected value… Quantitative outcomes are measured in
numeric terms; for example, dollars, hours, or turnover rates. Qualitative outcomes
are measured in non-numeric terms, which are often monitored through in-depth
interviews, direction observation, and/or written documentation.”
 References to use of a balanced scorecard of measures - Balanced scorecard – OGC
2005 (GSR1), P3M3 (GSR13), Kelly et al (GSR26), US VMM (GSR47)
Monetising benefit measures is a theme explored in CBA/Business Case Guidance (HMT Green
Book (GSR20); IPA 2017 (GSP19); Better Business Cases guidance (GSR33)); SROI (GSR27); White
House Circular No. A-94 Revised (GSR40) and other sources. These publications typically
recommend expressing all benefits in financial terms using market prices and contingent
23
valuation/revealed preference methods for non-financial benefits. Similarly, MSP (GSR2)
states, “best practice is to express benefits in financial terms wherever possible”. Other sources
disagree and warn about measuring non-financial benefits in financial terms for benefits
realisation management purposes:
 Bradley (GSR6) argues that giving financial values to non-financial benefits is

dangerous and should only be done where there is “a real flow of money”
 NSW BRM Framework (BSR52), “Care should be taken if trying to give financial
values to non-financial benefits as this can be misleading if the value is used as part
of an investment justification, but will not be able to be realised and measured”
 NZ Guidance (GSR59) “Although many benefits can be quantified financially, there
are others where it is difficult, undesirable, or insensitive to attribute a financial
value…Often non-monetary benefits can be monetised using CBA techniques such as
those in the CBA guidance. While this is an excellent technique for being able to
compare projects across varying outcomes to aid decision makers, this monetised
outcome should not be used for benefit realisation purposes as there is usually no
associated cash flow impact.”
Linking benefit measures to Organisational KPI’s/Strategic Objectives is recommended in

several sources, emphasising the importance of linking benefits measures to strategic
objectives and organisational KPIs. For example:
 MSP (GSR2)Benefit Profile includes –“KPIs in the business operations that will be
affected by the benefit”
 MoP (GSR3) refers to “development of a consistent set of metrics to link benefits to
strategic objectives”
 PRINCE2 (GSR7) - Benefits should be aligned to corporate objectives.
Reference to use of standard measures is made by:
 MSP (GSR1) – in the Benefits Management Strategy

 MoP (GSR3) - Benefits eligibility rules including a consistent portfolio-wide approach
to benefits quantification and valuation
 NSW (GSR52) Government are currently developing a Measures Dictionary for the
NSW Government Digital Strategy
 NZ Government Outcomes Catalogue Tool (Additional Note under GSR61) includes
measures for a variety of social outcomes - “The Treasury and Superu have
developed a catalogue of the social outcomes for government priority programmes.
The catalogue maps government priorities, outcomes, and proposed or actual
measures (where available) for a range of social sector programmes. It also lists how
24
the data is collected, when and by whom. This catalogue is extremely helpful when
developing the benefits for new social investments”
Other detailed practical points on benefits measurement include
 Use existing information sources or performance measurement systems - OGC 2005

(GSR12)
 Use of SMART measures – HMT Green Book (GSR20); Better Business Cases guidance
(GSR33); NSW guidance (GSR51); Western Australia (GSR56); and NZ (SMAART) (GSR59)
 Adjusting benefit forecasts:

o HMT Green Book (GSR20) adjust for leakage, deadweight,
displacement/substitution, distributional impacts, relative price movements, and
optimism bias;
o SROI (GSR27) – adjust for ‘Deadweight’, Displacement’; ‘Attribution’ and ‘Drop
off’ over time.
 Use of a normalised scale – the VMM (GSR47) combines quantitative and qualitative
metrics using a normalized scale in calculating the value score – “the normalized scale
provides a method for integrating objective and subjective measures of value into a
single decision metric.”
 Aus Fed Govt 2012 (GSR46) – suggests the use of proxy measures for qualitative
benefits.
Qu 6. What happens with a gap between benefits accountability and project

implementation?
The government body literature on measurement/forecasting accuracy and benefits realisation

measurement in practice identifies:
 Business Cases do not always clearly identify the benefits and when they do they are
commonly over-stated due to optimism bias and strategic misrepresentation. This issue
is one of the root causes of a subsequent failure on accountability later on; and
 A widespread and on-going failure to measure benefits realisation – particularly after
project/program closure, reflecting the gap referred to in this question.
25
4.2 Professional Body Sources
Q1: What is the “state of the art”in measuring benefits? (Normative/descriptive, Project,
Of the 42 sources reviewed, 32 were exclusively or primarily normative, i.e. what should be the
case, and 10 included a mixture of normative and descriptive content. None were purely
descriptive of current practice without any element of guidance. In the case of the PMI Thought
Leadership reports (PBR 15-22) major surveys of current practice were used to inform
recommendations for change, whereas other publications used case studies and examples
alongside guidance (for example APMG Managing Benefits, PBR 27). It should be noted that
some documents, particularly PMI Practice Guides, are not focused primarily on benefits.
In the professional bodies literature, the focus of the guidance will sometimes be specific to
one level - projects, programs or portfolios – or it may be concerned with the respective
responsibilities between these levels and the wider organisation. It is noteworthy that ISO
Standards now cover the full range of project/program/portfolio management levels and also
their integration. Table 4.1 indicates that many of the publications have a ‘joining up’ focus,
while many others are concerned with either projects and programs or solely with projects.
Table 4.1 Focus of guidance
Combination of responsibilities for BM/BRM No. of % of

publications publications
Project, Program and Portfolio, Wider organization 16 38
Project and Program only 8 19
Project and Portfolio only 1 2
Program and Portfolio only 2 5
Project only 7 17
Program only 2 5
Portfolio only 2 5
Wider organization only 1 2
Not specified 3 7
Total 42 100
26
Some sources made specific reference to responsibilities for BRM at the project, program and
portfolio level (for example, PMI Managing Change Guide, PBR6), while in others projects and
programs are generally bracketed together when making recommendations for BRM (for
example, PMI Thought leadership reports, PBR17 and PBR18; APM research fund report on
Social Cost-benefit Analysis, PBR, 26). Some sources adopt the position that benefits are usually
generated through programmes,
- ‘large projects can deliver benefits but most projects deliver capabilities, so benefits
usually arise from the roll out of a programme’ (APM SIG White Paper ‘Social Return on
Investment (SROI), PBR25, p6).
In the change management professional body literature the term ‘change initiative’ is favoured,
rather than projects, programs and portfolios (see PBR28, PBR29, PBR34). Some sources
emphasize the importance of the portfolio level for managing benefits, and achieving alignment
of strategy and investment in change. For example Val IT’s first guiding principle is that IT
enabled investment will be managed as a portfolio, and portfolio management is one of the
three domains for the application of the set of Val IT principles (PBR31), while the International
Centre for Complex Project Management emphasises a portfolio, rather than a project
perspective on benefits management (PBR34).
Although most of the recent professional body sources pay attention to the levels at which
BRM is to be focused, there is less on the subject of measurement of benefits. Some sources
highlight the importance of measurement, for example,
- ‘In addition, the approach requires attention to metrics. Part of the conversation when
benefits are identified is how the organization will know whether it’s on track to
achieving them. As central components of benefits realization management, metrics and
benefits tracking help to define objectives and critical success factors and to determine
if—and how—they are achievable’, PMI Pulse of the Profession report on strategic
impact of projects (PBR15) p7.
However, there are few sources which provide practical advice on measurement. One of the
few examples is the APMG Managing Benefits guide (PBR27), which includes measurement
issues in the barriers to benefits management in chapter 4 and provides guidance on selecting
measures in Chapter 7.
Q2. At what point(s) in the project are outcome benefits measures developed, defined and
selected?
28 sources out of 42 made some reference to this.
27
There is a consensus that benefits measurement is an activity that needs to be commenced
early on in the life cycle of a project, program or portfolio, to provide a baseline for monitoring
later on, and measures are therefore included in the business planning documentation. For
example, the PMI Thought Leadership reports (PBR15-22) all have ‘identify benefits’ as the first
of the three phases in the benefits life-cycle. The ‘Strengthening benefits awareness in the C-
suite’ report (PBR19) includes a recommendation to ‘Embed BRM in strategy making and
portfolio management from the start’ (p19).
In many of the publications, setting benefits measures is part of a second stage or phase in a
process. For example, the APM Body of Knowledge (PBR23) has ‘Identify and structure benefits’
as the second stage in the benefits management process, including benefits mapping (P124-
125). It follows on from ‘Benefits Management Plan’ where the overall approach is set out. In
ISACA Cobit 5 (PBR32) metrics for Benefits Delivery at the portfolio, program and project level,
(as well as at the operational level) are established in EDM02 Ensure Benefits Delivery,
following on from EDM01 Ensure Governance Framework Setting and Maintenance.
Some sources make explicit the gradual development of benefits measures as the nature of the
project or program becomes clearer over time, linked with governance decisions. For example,
BS ISO 21503: 2017 Guidance on program management (PBR40) states that
‘Benefit identification and analysis should begin when the programme is being
considered. After the programme has been established, a more detailed set of benefits
to be realized should be identified, analysed and prioritized. Benefit identification and
analysis may include, but are not limited to, the following: — identifying expected
benefits; — identifying benefit owners for each benefit to be realized; — aligning
benefits with strategic and other objectives; — defining performance metrics and
reporting for each benefit; - determining time frames for benefit realization’ p.13.
APMG Managing Benefits (PBR27) separates out the identification and quantification of
benefits from their subsequent appraisal. Baselining and forecasting occurs at the beginning –
‘Start benefits tracking as early as possible during development and delivery so that data
against which to measure benefits realization is available’ p73. Then the ‘Value & Appraise’
practice involves valuing benefits in monetary terms for the Business Case.
28
Focus on outcome and/or intermediate benefits?
In reviewing this question, the relationship between the term ‘outcome’ and the term ‘benefit’
requires exploration, as they are sometimes viewed as synonymous, and sometimes as distinct
from one another. As highlighted in our 2016 report to the PMI on BM/BRM terminology, a
number of professional body sources define benefits as outcomes with certain characteristics,
such as
 PMI, The Standard for Program Management, 2013 (PBR3) defines benefits in outcome
terms: “An outcome of actions, behaviors, products or services that provide utility to the
sponsoring organization as well as to the program’s intended beneficiaries” (p165).
 The ISACA Glossary also defines benefits in outcome terms: Benefit - “In business, an
outcome whose nature and value (expressed in various ways) are considered
advantageous by an enterprise.”
On the other hand, the PMI, Managing Change in Organizations: A Practice Guide, 2013 (PBR6)
states that ‘Capabilities lead to results, lead to outcomes, lead to benefits’ p73.
The term ‘outcome benefit’ did not appear to be used, but there were 10 sources out of the 42
which distinguished between different types of outcome or different types of benefit. ISACA Val
IT (PBR31) referred specifically to different types of outcome,
‘Utilise appropriate methods and techniques, involving all key stakeholders, to develop
and document a complete and shared understanding of the expected business outcomes
(both intermediate, or lead, and end, or lag, outcomes) of the candidate programs, how
they will be measured, and the full scope of initiatives required to achieve the expected
outcomes’ (Part of IM2.1 Develop a clear and complete understanding of the candidate
program).
Other sources also used the dualities ‘intermediate/end’ and ‘lead/lag’, but applied them to
benefits. For example
 The Change Management Body of Knowledge – Change Management Institute, 1st

Edition (2013) (PBR28) refers to “The path from enabling changes through ‘intermediate
benefits’ to ‘end benefits’”. p61.
 The APM Competence Framework, 2nd edition (2016), Competence 27, Benefits
Management (PBR24) includes under Knowledge Area 6 ‘measures for both leading
and lagging realisation indicators’, P32.
29
As a slight variation, the APM SIG White Paper ‘Social Return on Investment (SROI) – a powerful
tool for the realisation of benefits’ (PBR25) contrasted ‘intermediate’ with ‘ultimate’ outcomes,
and gave a practical example,
‘Once the SROI framework has been established, it is relatively straightforward to follow
through the causality chain, and measure at key points. This means that the business-as-
usual team can measure how well they are doing against a particular desired result,
either using the ultimate outcome (for example, people not using hospital for smoking-
related conditions), or a proxy measure (sometimes termed intermediate outcome) that
is more appropriate at the time of measurement (the number of people smoking, and
the amount they smoke)’ P 15.
As another variation, the PMI, Managing Change in Organizations: A Practice Guide, 2013
(PBR6) refers to ‘intermediate and tangible benefits that are linked to the ultimate purpose, as
well as the pace of delivery, are set during scope delineation’ (p88).
Other sources distinguish between short and long term benefits. The PMI, Pulse of the
Profession report ‘The Strategic Impact of Projects: Identify benefits to drive business results’
found that ‘Only one-third of organizations report that they differentiate between short-term
benefits (those expected shortly after project completion) and long-term benefits (those
expected months or years later)’ (p14).
The SROI study quoted above refers to the causality chain as a mechanism to link different
types of benefit, and a variety of benefits mapping and benefits dependency tools are used in
the literature to draw distinctions between different benefit types and how they relate, on the
one hand to strategic objectives, and, on the other hand, to the activities being undertaken.
APMG ‘Managing Benefits’ guidance (PBR27) summarises a number of different approaches to
benefits mapping, which are associated with sources covered in other literature reviews, such
as
- The MSP benefits map (Government Bodies review)
- The Results Chain (Consultant/Practitioner review
- The benefits dependency network (Academic literature review).
One of the benefits mapping tools not covered elsewhere in this report is the ‘benefits logic
map’, which distinguishes between intermediate benefits and end benefits. The APMG
‘Managing Benefits’ guidance (PBR27) refers to an example of the use of this technique, for the
‘Tell us once’ program, with intermediate benefits mainly falling to public authorities and end
benefits mainly benefiting citizens (p65).
There are, therefore, different dualities used to categorise benefits but in the professional
bodies literature intermediate/outcome is not used. The terms that have been found are
30
intermediate/end, intermediate/ultimate, lead/lag and short term/long term. There are
different ways in which distinctions are made, including
- By importance, in the eyes of beneficiaries or other stakeholders
- By relationship to strategic objectives
- By time of occurrence as to when they happen
- By time as to when they can be measured to monitor progress.
Often the above factors are related to one another in practice, but they are all slightly different
in terms of their implications and the interdependencies implied. In a benefits map, the
interdependencies are shown, but there can be wider issues, in terms of the attribution of
benefits, illustrated by the following excerpt from the Change Management Institute, The
Effective Change Manager’s Handbook (2015) (PBR28),
‘There is often a chain of benefits with intermediate benefits linked to final or end
benefits… From the perspective of tracking benefits realization, the point to note is that
monitoring leading measures of intermediate benefit achievement provides evidence to
confirm that end benefit realization can be attributed to a specific change initiative’
(p165-166).
In some cases, there is not necessarily any interdependence between categories of benefit. For
example, short and long term benefits are not necessarily causally connected, and if they are,
the question of degree of attribution will arise. None of the sources explored such issues in any
depth.
Linking to specific stakeholders
Many of the sources make general statements about the importance of stakeholder
engagement in benefits measurement, in terms of, for example,
- choosing measures that are meaningful for stakeholders (ICCPM - Complex Project
Management Global Perspectives and the Strategic Agenda to 2025, Compendium of
working papers, (PBR34, p154),
- maintaining communications with key stakeholders and optimising benefits for
stakeholders (PMI, Pulse of the Profession report – Beyond the project : sustain benefits
to optimise business value (PBR17))
- reconciling the views of stakeholders on value (The European Standard EN 12973,
(PBR35, p5).
Some sources go into detail on the engagement of beneficiaries. APMG ‘Managing Benefits’
(PBR27) includes guidance on customer and staff surveys, with two case studies, of ‘BT and the
RiO system’ and the ‘Tell us Once programme’. APM SIG White Paper ‘Social Return on
Investment (SROI) – a powerful tool for the realisation of benefits’(PBR25) indicates how
31
benefits measures in SROI are derived from stakeholder interviews. A case study of the use of
SROI is used, with the different stakeholder groups listed. APM Research Fund Series ‘The
importance of conventions: a critical evaluation of current practice in social cost-benefit
analysis’ (PBR26) contends that the interests of stakeholders are highlighted in the notion of
conventions. The meanings attached to particular measures/indicators and the moral
convictions driving those meanings are often not considered (p8).
Most of the 42 sources had something to say on this question, with only 13 not addressing it.
Amongst the other sources, there are a variety of different roles identified, without any
dominant individual being highlighted across the literature. Professional bodies with
responsibilities across the project, programme and portfolio domains go some way to
identifying a coherent approach across the whole organisation, although this can be hampered
when guidance documents have been produced at different times, as is the case, for example,
for the ISO Standards.
For the PMI, the ‘Governance of Portfolios, Programs and Projects Practice Guide, 2016’ (PBR7)
combines the roles in the Standards for Portfolio Management and Program Management
(PBR3 and 4), in ascribing key roles to the Portfolio Manager and Program Manager in assessing
the benefits. The PMI, Pulse of the Profession report – ‘Beyond the project : sustain benefits to
optimise business value’ (PBR17) provides a complementary perspective emphasizing the
transfer of projects. Various benefits-related activities at project transfer to BAU are listed and
responsibilities for them identified. The project manager and business/benefit owner feature
strongly in terms of responsibilities. However, the recommendations of the report strongly
recommend cross-functional responsibilities and ownership (P17).
For the APM, the Body of Knowledge 6th edition (PBR23) specifies that under programme
management, the business change manager is responsible for successful transition and benefits
realization (p14). The project or programme sponsor has ultimate accountability for benefits,
but the project or programme manager will be responsible for preparing the business case
(p94).
32
APMG ‘Managing Benefits’ (PBR27) provides guidance on the various roles which may be
required to collaborate on identifying benefits measures
‘The usual approach to identifying benefits measures is via a facilitated workshop

including: the Benefit Owners and end-users/customers; the Portfolio Benefits Manager;
Business Change Managers; representatives from the performance management
function (to advise on what current measures may be appropriate); and those who will
be responsible for collecting the data in due course including the Project/Programme
Office Benefits Manager.’(P105).
Following the identification of benefits, the assessment of benefits may be split between many
different individuals. BS ISO 21503: 2017 Guidance on program management (PBR40) advises
that ‘Benefit owners should be identified for each benefit to be realized’ (p13), while the
programme manager role includes coordination for achieving programme objectives and
realizing programme benefits (p8).
In some instances, professional bodies are using management tools to provide detailed
guidance on roles. For example, for each Val IT process (PBR30-33), a RACI (responsible,
accountable, consulted and informed) chart is provided.
The RACI chart decomposes the process into a set of key activities, indicating for each of these
activities who should be responsible, accountable, consulted and informed. For the Val IT
version 2.0 (PBR31) the roles and structures Included in RACI charts comprise :
- Board
- Business Sponsor (or Service Owner)
- Business unit managers/executives
- Compliance, audit, risk and security functions
- CEO
- CFO
- CIO
- Investment and services board (ISB)
- HR
- Program manager
- Program management office (PMO)
- Value Management office (VMO)
The activities in the RACI chart are derived from Val IT’s key management practices, but are
mostly defined at a more detailed level of granularity. Benefits are assessed at program stage-
gates specified by portfolio management for different categories of investments.
A major issue for professional bodies concerns the role of project managers for assessing
benefits. The AIPM Professional Competency Standards for Project Management – Certified
Practicing Senior Project Manager (CPSPM) – Jan 2016 v1.12 (PBR37) includes “Implements
appropriate metrics and measures in support of defining benefits” in the criteria for an
33
experienced Senior Project Manager (p.8), but in other cases assessing benefits is pitched at the
program and portfolio levels.
Accountability for benefits is often separated from responsibilities for measuring and
monitoring them. The PMI Thought Leadership Series - Establishing Benefits Ownership and
Accountability report (PBR20) found that
‘more mature organizations have a single person accountable for BRM for each initiative
or group of initiatives—24 percent versus 13 percent of organizations with low
maturity……since many anticipated benefits are not seen until after the project is
delivered, having one owner for ongoing benefits measurement and validation is critical.
It puts someone in charge of consciously and deliberately monitoring and measuring
benefits. This role can fall under many names—business owner, business manager,
program director, divisional head, or product manager. P4.
A specific high level structure is proposed by the PMI Thought Leadership Series - Benefits
Thinking Movement report (PBR22), which suggests,
In the Identify phase: ‘Consider the role of a Chief Benefits Officer (CBO) to set the
policies, procedures, and measurement of benefits. This role could be fulfilled by an
existing Chief Strategy Officer or EPMO equivalent body. The governance body forms a
Benefits Alliance that:
• Reviews the business cases for all strategic initiatives
• Approves the identified benefits and proposed measurement methods
• Drives benefits culture in collaboration with end-user business units’ (p2)
18 sources address emergent benefits explicitly or implicitly, while 24 sources do not consider
or plan for this eventuality. However, of the 18 sources, very few specifically refer to new
measures. In most cases emergence is referred to as a general feature of P3M, or a similar term
such as unplanned or unintended benefits is used.
APMG ‘Managing Benefits’ (PBR27) is one of the few sources that provides a definition in its
glossary. It defines emergent benefits as “Benefits that emerge during the design, development,
deployment and application of the new ways of working, rather than being identified at the
34
start of the initiative.” p245. Approaches to identifying them include the ‘Scout and Beacon’
approach (p113).
The Change Management Body of Knowledge – Change Management Institute, 1st Edition
(2013) (PBR28) refers to “Identifying new and unexpected benefits or negative effects that
might emerge” (p61).
The PMI Thought Leadership Series- Benefits Realization Management Framework (PBR21)
specifies questions within the ‘Execute’ phase which reflect different implications of
emergence, including
 Are benefits frequently modified to reflect the most current information regarding
changing business conditions?
 Is there a formal process to discover new benefit opportunities?
 Is the project or program still relevant based on what benefits can be realized against
unexpected events or changes to the benefits realization plan?
Usually the implications of an emergent approach for benefit measures are assumed rather
than spelt out. A source that specifically refers to new measures or metrics being added is the
PMI - Business Analysis for Practitioners A Practice Guide, 2015 (PBR9), ‘post release,
stakeholders may identify new metrics that tie in to the current or new objectives’ p172.
Other terms used which explicitly imply emergence include
 unplanned, for example, ‘The benefits of a portfolio management approach include

identification and realization of unplanned benefits to create additional value’, APM
Body of Knowledge 6th edition, APM, 2012 (PBR23) P17.
 unintended, for example, unintended outcomes are part of the 3 E’s model in the APM
Research Fund Series ‘The importance of conventions: a critical evaluation of current
practice in social cost-benefit analysis’ (PBR26).
 The PMI, A Guide to the Project Management Body of Knowledge, (PMBOK®) 6th Edition
(PBR2) refers to to the Benefits Management Plan being maintained as an "iterative
activity"
 additional benefits, for example, the APM Competence Framework, 2 nd edition (PBR24)
states that ‘Benefits management refers to all of the activities devoted to ensuring that
the benefits intended from change initiatives, and any additional benefits that could
result, are achieved’.
Some sources specify processes which can be interpreted in different ways, which may or may
not result in additional measures. Examples are
35
 ISACA – VAL IT version 2.0 (PBR31) refers to the need to update the business case
throughout the life of a program, including any changes affecting projected costs and
benefits. However, this leaves it open whether this covers changes to existing measures
or whether it extends to consideration of new measures.
 BS ISO 21504:2015 Guidance on portfolio management (PBR41) refers under section
5.8.2 Optimizing portfolio components, recommends that the portfolio manager should
‘continuously analyse and improve the realization of benefits from the portfolio
components including reviewing success criteria’.
 The PMI, A Guide to the Project Management Body of Knowledge, (PMBOK®) 6th Edition
(PBR2) refers to to the Benefits Management Plan being maintained as an "iterative
activity"
Measurement post-project?
25 sources make no mention of measurement post-project. The 17 sources which do consider

this issue vary in the extent to which it is emphasised in the guidance. At the minimum, is the
acknowledgement that benefits are realised after the implementation of a project or program
has been achieved. For example, BS ISO 21503: 2017 Guidance on program management
(PBR40) specifies that ‘Benefits may be realized during the programme, at the end of the
programme, or after the programme has closed. Before the end of the programme, the
responsibility for the realization of benefits may be transferred to a new owner’ P13.
The transfer to Business As Usual (BAU) is referred to by APM Body of Knowledge 6th edition,
APM, 2012 (PBR23), which specifies that the bulk of the benefits may only be realised after the
project or programme has been completed. Therefore, long term actions and monitoring for
continued realisation should be part of the handover to BAU (P125). In relation to how long
‘long term’ means, the BoK indicates that after handover to BAU several benefits reviews may
be required, depending on when benefits are due to be realized.
The theme of short term and long term benefits is one which was touched on under Question 2.
As referred to there, many of the PMI Practice Standards and Thought Leadership reports
include this distinction (see PBR7, PBR9, PBR15). In PMI, Pulse of the Profession report –
Beyond the project : sustain benefits to optimise business value (PBR17), one of the case
studies refers to an engagement internally to integrate with and support business goals over 1-
2 years (P14).In the ‘sustain’ phase of the benefits life-cycle in PMI Thought Leadership Series-
Benefits Realization Management Framework (PBR21) there is guidance is optimizing benefits
on a sustainable basis, but nothing specific on how long to assess benefits, perhaps reflecting
the difficulty of providing generic guidance when context is so significant.
One source which is specific on the timescales for benefits measurement is the APM SIG White
Paper ‘Social Return on Investment (SROI) – a powerful tool for the realisation of benefits’
(PBR25) which suggests that the timeframe for SROI and the range of benefits measured
depends on the type of investment. It suggests 15 years for engineering projects with a 30-100
36
year lifetime. 18 months – 3 years may be appropriate for projects working with adults with
learning disabilities, because of changes in government policy over time (Para. 2.1).
The IT-enabled change sources refer to the concept of the ‘full economic life-cycle’ in
addressing the issue of benefits realization in the longer term. ISACA Val IT version 2.0 (PBR31)
defines the full economic life-cycle as ‘The period of time during which material business
benefits are expected to arise from and/or material expenditures (including investments,
running and retirement costs) are expected to be incurred by an investment program’. The
guidance in this source on ‘IM10.1 Retire the program’ is worth quoting at length, because it
addresses many of the issues affecting longer term benefits,
Retire the program from the active investment portfolio when all the projects within the
program have been completed and there is agreement that the desired business value
has been or has a high potential of being realised. Ensure that the program is brought to
an orderly closure, including formal approval of retirement by the ISB and the business
sponsor. Review and document lessons learned. Once the program is retired, it should be
removed from the active investment portfolio. Program retirement recognises that the
major activities planned to create value have been completed, but benefits monitoring,
realisation and optimisation will still need to be monitored and managed until the full
value of the program is realised and the changes have become ‘business as usual’. Even
at that stage, when the program results in an ongoing service or other assets or
resources, accountability and processes should be put in place to ensure that the
enterprise continues to optimise business value from the service, asset or resources.
Additional investments may be required at some future time to ensure that this occurs.
Some sources suggest where the responsibility of post-implementation measurement should

rest. Examples are
 The PMBOK® 6th Edition, 2017, suggests that where benefits may be realized after
project closure the "portfolio management office (PMO), portfolio steering committee or
some other business function...should evaluate the success at a later date to determine if
the outcomes met the business objectives." p547.
 British Computer Society publications Hughes, B. (2008) Exploiting IT for Business
Benefit (PBR30) states that ‘Having a programme management structure that lives
beyond the lives of individual projects means that there are people who can monitor the
actual capture of the benefits. The programme manager, along with the business
change managers, can take action to make sure that the benefits are actually achieved’
P 154.
the customer is?
37
Some of the sources are directed at specific sectors or types of project, according to the remit
of the professional body. Examples are the Computer/IT enabled change professional bodies
(PBR30-33) and the International Centre for Complex Project Management (PBR34). Most of
the guidance is aimed at the project management profession and is cross sector. In some cases
the generic nature of the guidance is explicitly stated. Examples are
 BS ISO 21500:2012 Guidance on project management (PBR39) ‘This International
Standard provides guidance for project management and can be used by any type of
organization, including public, private or community organizations, and for any type of
project, irrespective of complexity, size or duration’.P1.
 PMI, Organizational Project Management Maturity Model (OPM3®) 3RD Edition, 2013
(PBR5) is aimed at any organisation wishing to adopt OPM3.
Where guidance includes program and portfolio management it is usually indicted that it will
require tailoring to the organisational context. Therefore, BS ISO 21505:2017 Project,
Programme and Portfolio Management: Guidance on Governance (PBR42) specifies that the
guidance is generic but may need tailoring to the specific needs of the organisation.
Sometimes advice is provided to smaller organisation as to how they can tailor the advice to
their context. ISACA Val IT version 2.0 (PBR31) specifies that
The guidance and examples presented in Val IT are applicable to all enterprises and
address all aspects that should be contained in defining, evaluating, selecting and
managing any IT investment. This guidance, however, is not intended to be prescriptive,
and should be tailored to fit the enterprise’s management approach. Small and medium-
sized enterprises can adapt the templates and make them simpler to create and
maintain, but in all cases the model adopted should cover business alignment, cost and
benefits (financial and non-financial) and risks since these play a major role in every
investment analysis for every enterprise.
Where the professional body sources include descriptive information, in the form of survey
results or case studies, the implications for practice and for future guidance will be affected by
the organisations studied. The PMI Thought Leadership reports that included surveys, such as
Sources PBR15-17, mainly covered large organisations, with IT being the most common sector,
at around 15-17% of respondents.
The APM SIG White Paper ‘Social Return on Investment (SROI) – a powerful tool for the
realisation of benefits’ (PBR25) included a case study example of the ‘National Specialist Family
Service’, a charity helping people to manage drug and alcohol addiction and contribute to
society. It is claimed that SROI is ‘the measurement method of choice for non-profit projects and
programmes such as government and community and voluntary services (including charity, third
sector and public sector)’. However, it is also suggested that SROI is useful for other sectors, eg
measuring impact of CSR programmes for prIvate sector organizations (Executive Summary).
38
10 sources did not refer to the type of measure. Of the remaining 32, almost all, 31 in each
case, included both quantitative financial and quantitative non-financial measures. 21 included
reference to qualitative measures.
APMG ‘Managing Benefits’ (PBR27) includes a wealth of guidance on types of measure, with
many examples to illustrate different measurement approaches and issues. It states that
“Effective management of benefits realization is aided by the selection of appropriate measures
– at least one for each benefit, and preferably a suite of measures (including leading and
lagging measures, proxy indicators, evidence events, case studies, surveys and stories) to create
a ‘rich picture’ providing feedback on benefits realization from multiple perspectives” (p8).
Types of measure are categorized as quantitative or qualitative, financial or non-financial,

leading or lagging and direct or indirect (proxy), with examples given of each (Table 7.2, p106).
A benefits measurement taxonomy is provided, linking benefits category, measures and
indicators and measure/indicator type (Table 7.3, p107). An example of measures related to the
different types of financial benefit at Openreach (a BT company) is provided (Table 7.4, p108).
The benefits tracking and reporting section (from p126) includes guidance on presentation of
benefits information, using a number of examples, and techniques to aid benefits monitoring
(p130).
The different ways of categorizing measures used in APMG ‘Managing Benefits’ (PBR27) are
found in many other professional body sources, with an additional distinction made between
tangible and intangible. PMI, Pulse of the Profession report – Beyond the project : sustain
benefits to optimise business value (PBR17) draws the distinction between tangible and
intangible as,
Tangible benefits are easily measured and quantified—and are most commonly
associated with revenue and ROI. They also reflect cost savings, productivity, and
process improvements.
Intangible benefits cannot be easily quantified or measured, if at all. They typically give
insight into stakeholder relationships and factors that can impact revenue and ROI
indirectly. They include measures of customer satisfaction, risk mitigation,
organizational reputation, compliance, and brand equity or market share (p15).
39
The distinction between quantitative and qualitative is therefore similar, but not the same, as
the differences drawn between tangible and intangible.
Some sources suggest a hierarchy of benefits. For example, British Computer Society
publication, Hughes, B. (2008) Exploiting IT for Business Benefit (PBR30) classifies benefits as
 Quantified and valued (i.e. in monetary terms)

 Quantified but not valued
 Identified but not quantified (p135-136).
The disciplines of investment appraisal and performance management include many techniques
for conversion of measures from non-financial to financial and qualitative to quantitative. APM
Research Fund Series ‘The importance of conventions: a critical evaluation of current practice in
social cost-benefit analysis (PBR26) summarises some of the methods used - contingent
valuation, hedonic pricing, travel cost method, shadow pricing, quality-adjusted life year and
life satisfaction assessment – but maintains that there are some benefits which are very difficult
to quantify, such as the public amenity of green spaces (p6). Furthermore, ‘conventions of
quantification’ are underpinned by the moral values and social expectations of those who then
decide how to quantify social phenomenon (p8). The report acknowledges that this issue even
applies to approaches which explicitly aim to incorporate stakeholder perspectives and take a
broad view of benefits, such as the Social Return on Investment (SROI) framework.
Where sources refer to the use of qualitative benefits, this may be as a preliminary scoping
exercise, with the derivation of measures to quantify the benefits taking place later in the
benefits life-cycle. This might take the form of a survey asking stakeholders to rate statements
on a scale (APMG Managing Benefits, PBR27, p134 for an example of a customer survey).
However, it will not always be possible to obtain data which exactly reflects the qualitative
benefit. For example, the public amenity of green spaces has a variety of different benefits for
citizens, some of which are very difficult to measure, such as the aesthetic value of green
spaces in urban areas. The difficulties in finding appropriate quantitative measures for
qualitative benefits leads to the use of ‘proxy’ or indirect measures (APMG Managing Benefits,
PBR27, p106). Alternatively, qualitative research methods might be used to generate
supporting evidence, such as detailed case studies and stories. However, the PMI Thought
Leadership Series- Strengthening benefits awareness in the C-suite report (PBR19) found that
organizations find such issues challenging, ‘….very mature organizations are more likely than
the immature to cite measurement difficulties as barriers to BRM, in particular, those around
intangibles (45% to 25%, see Figure 13). This disparity could result from the fact that very
40
mature organizations are more aware of the problems and are struggling more intensely to
resolve them (p15).
Some professional body sources explicitly express a preference for financial measures. For
example, the APM Body of Knowledge 6th edition, 2012 (PBR23) states that , For programmes
intangible and non-financial benefits may be included in the business case. Caution should be
used to prevent giving them too much weight, because of their subjective nature (p175).
Most of the sources use the term measure, but some also refer to metrics. The terms may be
used synonymously, but where a distinction is drawn, the term ‘metric’ refers narrowly to data
whereas measure is a broader term. Thus BS ISO 21504:2015 Guidance on portfolio
management (PBR41) states that ‘Portfolio-specific metrics should be established, addressing
schedule, technical and financial performance………Portfolio-level measures should track the
overall health of the portfolio, value creation and benefits realization’ p10.
There are different views expressed as to how many measures/metrics should be tracked. PMI
Thought Leadership Series- Connecting business strategy and project management: Business
realization management (PBR18) states that ‘Some may think that developing KPIs adds a layer
of bureaucracy, yet BRM requires only a minimally sufficient set of metrics (typically one to
three) that focus on lead indicators of project performance, risks and interdependencies, and
value delivery. These metrics—along with regular updates (typically once every two to four
weeks)—link individual projects back to strategy and create transparency’ (p6). On the other
hand, some sources suggest a much higher number of measures. APM SIG White Paper ‘Social
Return on Investment (SROI) – a powerful tool for the realisation of benefits (PBR25) suggests
that ‘Benefits identified through stakeholder engagement to be narrowed down to perhaps 20-
30’. Related to this is the question as to whether indicators which are already collected by the
organization can be used to measure benefits. PMI - Business Analysis for Practitioners A
Practice Guide, 2015 (PBR9) refers to categories of KPIs generally collected by organizations,
such as finance and customers, and suggests that project goals will be associated with these
KPI’s, leading to the identification of metrics, but there may be additional metrics and
acceptance criteria, some of which are about the solution and some on project execution (see
p162-164).
The number of measures that it is appropriate to use might vary between different sectors. The
PMI Governance of Portfolios, Programs and Projects: A Practice Guide, 2016 (PBR14) suggests
that the private sector may focus more on quantitative financial benefits and other sectors on
all types more evenly (p47).
The PMI Thought Leadership reports which involved surveys of obtained information on the
frequencies with which particular measures are used by their participants. For example, the
41
PMI, Pulse of the Profession report – Delivering value: focus on benefits during project
execution (PBR16) found that the the highest frequencies for metrics used to measure and
track performance were for
 ‘customer or user satisfaction, customer retention or loyalty’ and

 ‘ROI, efficiency of operations, margin improvement, revenue generation, share of
market’ (p19).
PMI, Pulse of the Profession report – Delivering value: focus on benefits during project
execution (PBR16) also found 76% said that it was very or extremely important to identify
intangible project benefits, but only 38% said that their organisation routinely quantifies them.
The nature of measures identified as important can depend on the topic area for the
publication. Thus PMI, Managing Change in Organizations: A Practice Guide, 2013 (PBR6) places
emphasis on measures for the acceptance and adoption of change initiatives. This report is one
of those which also holds that not all measures of success can be translated into financial
terms, and an example of a flawed measure of this type is provided (p75).
Qu. 6. What happens with a gap between benefits accountability and project
implementation?
The analysis under this literature review which is most pertinent to this issue is the discussion
under Q3, on ‘who assesses the benefits’. In terms of addressing the gap between benefits
accountability and project implementation, the analysis in the PMI Thought Leadership reports,
‘Establishing Benefits Ownership and Accountability’ (PBR20) and ‘Benefits Thinking Movement’
(PBR22) is relevant. The first report found that mature organisations in BRM make individuals
accountable for benefits realization for specific initiatives. The second report put forward an
idea for a Chief Benefits Officer, heading up a Benefits Alliance which would prevent a gap
opening up.
42
4.3 Consultant/Practitioner Sources
Q1: What is the “state of the art” on benefits measures? (Normative/descriptive, Project,
Of the 24 sources reviewed, 22 were normative, i.e. what should be the case, of which 6 were
also descriptive, i.e. what is the case, with a further 2 being only descriptive. This suggests a
gap between what should be (the ideal “state of the art”) and what is (general practice).
There were an equal number of sources (8) that covered project, program and portfolio (P3M),
and project and program (PPM). One of the PPM sources (Curley, CPR3) referred only to IT
projects and programs. A number of sources, one quite specifically (Thorp, CPR24) go beyond
PPM to a broader Enterprise Value Management (EVM) view, while within the P3M group,
there was not always the same level of coverage of all three elements. Of the other sources,
one source covered portfolio and project, one only program, two only project, and one only
portfolio, while 3 made no reference to any of the 3 terms. It should be noted here that there
continue to be inconsistencies in the use of terminology in this space. Specifically, the terms
project and program are still occasionally being used interchangeably, while there a number of
different view of the relationship between benefits and value
There are a number of common themes throughout many of the sources:
1. Benefits do not come from technology in and of itself, but rather from the change that
technology both shapes and enables - change that must be both led and managed
(Bradley (GSR6), ITCMF (CPR10), Jenner (CPR6), Matharu & Green (CPR16), Moorehouse
(CPR18), Thiry (CPR23), Thorp (CPR24))
2. If not yet universal, as per the comment above, there is a strong view in this literature
that makes a distinction between project, program and portfolio, where Projects
deliver outputs/capabilities, Programs deliver benefits and outcomes enabled by those
capabilities, and Portfolios optimise overall value from the suite of programs and their
resulting assets (Davies & Davies (CPR4), ITCMF (CPR10), Thorp (CPR24))
3. As referred to above, there continues to be a significant gap between what should be (the
ideal “state of the art”) and what is (general practice), as evidenced in a number of the
sources:
a. . . …divergence in approaches to benefits realisation between project,
programme, change professionals and finance professionals. Recognising strong
finance representation, it is no surprise that a standard set of KPIs, ROI analysis
and NPV are the three most commonly used approaches. A significant number of
respondents have not considered using benefit maps (69%) or benefit profiling
(57%) (Deloitte, CPR5)
b. … in practice “Benefits are tracked through KPIs, even if they don’t fit” Reasons
for failure include - “dollars realisation as a proxy for benefits realization”;
“Project Sponsors will continue to ‘get away’ with sloppy benefits claims as long
43
as they know that no one will be measuring it (or even looking for it) when the
project is complete.” (Simms, CPR21)
c. Use of benefits profiles and key benefit indicators were found to be rare – in less
than 10% of organisations were they ‘normally used’, and “The BRM toolkit is
drastically underused” (Moorhouse, CPR18)
4. There is also one under-emphasised theme – accountability, a term which is discussed
in 3 sources (ITCMF (CPR10), Simms (CPR21), Thorp (CPR24).
Q2: At what point(s) in the project are outcome benefits measures developed, defined and
selected?
There is some overlap between this question, and Question 3 regarding when measurement
occurs. Observations here include:
 With regard to outcome measures, 8 sources specifically mentioned outcomes, while 3

implied them, and 6 mentioned intermediate outcomes/benefits, which are key to
understanding and monitoring interdependencies, while 3 implied them.
 While some expression of the desired outcomes of an investment are usually general
stated at in the initial stages of any investment, most of the sources identify the need
for additional analysis to provide greater clarity on the desired outcomes, and how they
will be measured.
 A number of sources (Bradley (GSR6), Davies & Davies (CPR4), EPMC (CPR6), Evans &
Cesaro (CPR7), ITCMF (CPR10), Jenner (CPR6), Kerzner (CPR13), Thiry (CPR23), Thorp
(CPR24) refer to the business case as the repository for measurement information,
either including, or being supported by Benefits Profiles, Benefits Registers, and/or
Benefits Realization Plans. Two of these sources (ITCMF (CPR10), Thorp (CPR24)) also
suggest a more pro-active use of the business case, going beyond “Traditional one-off
business cases included forecasts of end benefits only for the sake of justifying project
work” to “Review, update, and re-evaluate business cases dynamically, throughout the
full investment cycle, according to changing business conditions”
 The notion that benefits are defined at the beginning and measurement occurs at the
“end” still exists, and is inferred to by a number of sources. However, in today’s world of
increasingly complex investments, many of them having a significant technology (or
digital) component, with the technology itself also changing at an ever-increasing rate,
this is no longer the case. There is an increasing understanding that measurements will
evolve – some will be changed, or dropped, and new ones will be developed, defined
and added over the life cycle of an investment decision, which includes the on-going use
of the assets resulting from that decision (see examples under Question 3, emergent
benefits).
 Seven sources discussed stakeholder engagement in the development, definition and
selection of investment measurements, and their validation, a number of them
identifying the use of workshops to do this. The roles of specific stakeholders are
discussed in the summary of the next question.
44
 One source (Thorp, CPR24) stresses that “It is difficult, or impossible to come up with
relevant metrics without a clear understanding of the expected outcomes. Metrics must
clearly link the contribution of investments to outcomes, and themselves be linked to
clear lines of accountability”. In order to ensure that this is the case, the (Benefits
Realization) approach includes a benefits/value mapping technique, the Results Chain,
developed by DMR (now Fujitsu Consulting), which is used to help organisations to
prepare a comprehensive and accurate model of their benefits realization process. A
number of other sources also reference the use of some type of benefits/value mapping
technique (Bradley (GSR6), Davies & Davies (CPR4), ITCMF (CPR10)).
 Using a workshop approach with stakeholders, these benefits/value mapping
techniques can be applied to varying degrees to either define, surface, refine or clarify
investment outcomes (both intermediate / enabling and end outcomes), initiatives,
contributions, and assumptions, all of which should be measurable.
There are a large number of different roles described related to benefits measurement. The
roles identified in the Consultant/practitioner literature can be divided up into three broad
categories, in terms of level of seniority and responsibility/accountability:
1. Senior Executive and supporting roles, including:

a. Higher level governance board (programme or portfolio) (Evans & Cesaro, CPR7);
b. Investment Decision Board (Thorp, CPR24);
c. Value Management Office (EPMC (CPR6), Thorp (CPR24)).
2. Business Managers and supporting roles, including:
a. Business Sponsor/Senior Responsible Officer (Public Sector) (Multiple refs);
b. Business Value Program Office (Curley, CPR3);
c. Project Sponsor, Project Champion & End User (Melton, Iles-Smith & Yates,
CPR17);
Note that Bradley (GSR6) states that: “Business Unit Managers will be Benefit
Owners different from Measure Owners who, again, may be different from
Measure Monitors, with Benefits Facilitators, sitting outside programmes”.
3. Program & Project Managers & supporting roles, including:
a. Benefits Realisation Manager & Data Manager (Matharu & Green, CPR16);
b. Business Change Manager (BCM) (Evans & Cesaro, Letavec, CPR7);
c. Project Management Office (Kerzner (CPR13), Letavec (CPR15)); and
d. Benefits Coordinators (Payne, CPR19).
45
Regardless of what titles are used, ITCMF (CPR10) suggests the use of (RACI) charts to define
clear IT and business roles and responsibilities related to benefits realization and optimization
from individual IT-enabled change investments and resulting assets across the full lifecycle of an
investment decision.
Seven sources specifically discuss emergent benefits. Examples include
 Value Management Office charged with capturing emergent benefits and disseminating
learning’s and best practice.” (EPMC, p96 (CPR6))
 Benefits management includes ensuring “emergent benefits are captured and

disseminated, and capability and capacity created is leveraged to create additional
value.” (Jenner 2010 pxi (CPR11))
 “we go beyond realising forecast benefits to capture benefits as they emerge and create
value by exploiting capability and capacity on an on-going basis.” (Jenner 2011 p154,
CPR11)
 Emergent benefits are seen as opportunities in later stages to optimise benefits

realization (Letavec 2014 p7, CPR7).
 “If there are no emergent benefits identified from a change it is most probably because
the benefits management process is not working as it should.” (p94) and “identify the
measure and confirm the benefit owner” (Matharu & Green p95, CPR16)
Kerzner & Saladis (CPR14) imply that emergence will affect benefit measures, saying: “Metrics
can change over the duration of the project”.
Emergent benefits are identified as more understanding of the investment is gained, or in

response to changes to the internal or external environment (Evans & Cesaro (CPR7), ITCMF
(CPR10), Jenner (CPR11), Thorp (CPR24)). In 3 of the above cases, there is a correlation with the
use of benefits/value mapping techniques and/or “lead” and “lag” measures (link to Qu. 2),
which can indicate when such changes may be required. Thorp (CPR24) references the use of
pre-defined “decision points referred to as stage gates (at which) programs can be assessed,
continued without change, modified, delayed or even cancelled”
46
There is quite a range of opinions as to when assessment of benefits ends with 14 sources
either not mentioning it, or not making a specific enough reference.
 Two sources (ITCMF (CPR10), Thorp (CPR24)) that use the term “full life cycle”, say that
”there should be on-going regular reviews of the resulting assets and services to ensure
and assure that they continue to deliver benefits and contribute to business value”.
Thorp goes on to say that “The program can only be considered successfully ended when
all intermediate and end outcomes (which may have been modified over time as
knowledge is gained and the context changes) have been realized. Even then, there
should be on-going regular reviews of the resulting assets and services to ensure and
assure that they continue to deliver benefits and contribute to business value.”
 Similarly, Jenner (CPR12) in discussing the project life cycle (PLC), states that BM runs
“through project implementation, and beyond project closure to business as usual” – in a
later paper, he also introduces the term: “Across the BM cycle”.
 Kerzner (CPR13) also uses the term PLC but says that “benefits are measured during the
project and value at the end”, which is somewhat ambiguous. EPMC (CPR6) states that:
After deployment – “Benefits need to be actively managed – to ensure that forecast
benefits are realized (especially important where those benefits are dependent on
business change), and to capture benefits that were not anticipated at the Business Case
stage.”
 Curley (CPR3) focuses on post-implementation, saying that “Contrary to current practice,
there is likely to be more value in measuring ROI during and after implementation than
in estimating ROI prior to investment approval.”
 A number of other sources reference measurement post/after the project (Simms,
CPR21), or use terms that imply measurement post-project, including during “Benefits
Sustainment” (Letavec, CPR7), “Benefits Realization” (Matharu & Green, CPR16), and
“Operate” (Payne, CPR19).
development, etc.) or by industry, project size, potential social impact, or even who the
customer is?
Most of the guidance is cross sector although there is a split between (mainly) public sector or
private sector guidance. Some sources are aimed at ICT investments, and a couple of sources
have another specific focus. e.g. Education (CiJ Consultancy Services, CPR2) and Manufacturing
(Melton et al., CPR17). In terms of project size/type, most either as stated, or by implication
appear aimed at more complex programs/projects with a significant change component.
With regard to “project type”, 2 sources (Thorp (CPR24), ITCMF (CPR10) state that it is
important to realize that “one size does not fit all”, and to “recognise that there are different
categories of investments and assets, and organizational contexts that will require different
47
approaches (in terms of scalability and adaptation)” (ITCMF, CPR10). This should be handled by
portfolio management, where “portfolio management categorizes programs according to the
types of decision that the portfolio managers can make and the nature of the investment.
Programs vary in the degree of discretion an organization has in undertaking the program, the
program’s relative value (where value is a function of alignment, benefits, costs and risks), and
the relative difficulty of realizing that value (both delivering the necessary capabilities, and
adopting and using them such the value is realised). Examples of Program Categorisation
include:
 Mandatory
o Legal Requirement
o Parent Co. Requirement
 Sustain
 Development
 Growth
 Infrastructure
 Business Opportunity (Thorp, CPR24).
Within organizations, 8 sources are non-specific about the target audience, beyond which there
is a broad range of targeted audiences including:
 “Boards, executives, and senior business management, including the CIO” (Thorp,
CPR24);
 Senior Executives (Evans & Cesaro, CPR7)
 Senior management (Simms, Simms & Chapman, CPR22);
 Project sponsors (Simms, CPR21);
 Senior IT management (Gartner (CPR9), Jenner (CPR12));
 Portfolio Managers (EPMC); PPM Community (Evans & Cesaro, CPR7);
 BM Community (Aus Benefits Institute);
 PPM (Kerzner (CPR13), Matharu & Green (CPR16));
 Change Managers (Matharu & Green, CPR16); and
 Public servants (Jenner, CPR11).
19 sources covered both Quantitative & Qualitative. Examples of these included:
 Tangible or Intangible; Direct or Enabled; Financial or Economic (i.e. those with an

attributed monetary value); Cashable or Non-cashable; and Planned or Emergent
(Bradley, GSR6);
 Financial – ROI, operating margin; Future/strategic – surveys on image; Internal – time,
48
cost, scope, efficiency (Kerzner, CPR13);
 Financial; customer; competitive; capability; productivity and risk-reduction. (Simms,
CPR 21).
5 sources covered, or implied, Qualitative first leading to Quantitative stating:
 “Benefits and outcomes are initially described in qualitative terms, Measures quantify
this, to derive a financial value for the project and to enable tracking” (Payne, CPR19);
 “To manage for business value, an organisation must first create an awareness of the
intangibles and then figure out a way to translate these benefits into quantitative
terms.” (Curley, CPR3);
 Definition of measure is something that quantifies performance. Use of the Balanced
Scorecard for Qualitative analysis as a prelude to quantitative performance
management. (Davies & Davies, CPR4);
 Qualitative benefits are an initial step, leading on to quantification - there are 4 types of
financial benefit: sustainable financial benefits; one-off financial savings; financial cost
avoidance; and Increase in sales performance (Melton, Iles-Smith & Yates, CPR17);
 The key word in the description of Measurement is “Relevant”. Many organisations
measure a large number of things, but those few key metrics that are relevant to
creating and sustaining value, are lost in the “noise”. The essential point about
measurement is that, by definition, it involves quantification in some form. Again, the
Results Chain is the integrating technique in this approach. A simple rule is that each
outcome in a Results Chain must be described in a way that forces measurement, using
a phrase containing relatively precise language. The acronym MEDIC represents the
following:
 M: e.g. a level of service Maintained;
 E: e.g. a function Eliminated;
 D: e.g. turnaround time Decreased;
 I: e.g. revenue Increased;
 C: e.g. a certain capability Created (Thorp (CPR24)).
Of the quantified measures 17 were financial and 5 non-financial.
Seven sources also mentioned intangible benefits, with regard to which Jenner states
“(intangible benefits are) benefits that are difficult to quantify and measure reliably such as
improved staff morale and decision-making - in such cases proxy indicators of such benefits can
be developed.” (Jenner, CPR12)
6. What happens with a gap between benefits accountability and project implementation?
In respect of Qu. 1 'state of the art' it was mentioned that a critical element in benefits
measurement and management is accountability. This is emphasised by IT-CMF (CPR10),
Simms (CPR21) and Thorp (CPR24). Experience would suggest lack of clear and accepted
accountability for the realization of benefits/value is a significant and on-going issue in both the
49
public and private sectors. As the old adage goes, "you can’t manage what you can’t measure”,
but, without accountability, management is an illusion. Under Qu. 3, a number of roles at
senior level were referred to which would be accountable for benefits under the RACI
framework (IT-CMF, CPR10). Where there is a loss of focus on benefits during implementation,
one of the critical areas to investigate is the roles of these people.
50
4.4 Academic Sources (citations ‘AOR’ to Appendix 4 and ‘ANR’ to Appendix 4a)
In the academic sources which included material of relevance to benefits methods, there are a
number of different themes explored. There is a mix of normative, descriptive and mixed
sources (Table 4.2). The most frequent level of P3M addressed is projects, but many sources
also covered projects/programs or projects/portfolios (Table 4.3). The sources include
- Books written by academics on BRM. Amongst the books written by the pioneers of
benefits management, those included in the academic category are ‘Benefits
management: how to increase the business value of your IT projects, by Ward and
Daniel (AOR88), and ‘Achieving maximum value from information systems – a process
approach’ by Remenyi et al. (AOR74). Other ‘how to do it’ books in the academic
literature review are Project management for the creation of organisational value, by
Zwikael and Smyrk (AOR97) and ‘Benefits Realization Management: Strategic Value from
Portfolios, Programs, and Projects’ by Serra (AOR84). All these sources include material
on benefits measurement as part of their overall method for BRM.
- The Cranfield approach outlined in Ward and Daniel (AOR88) is the most widely adopted
method for BRM globally, and there are a number of sources which report on
applications of and/or further developments from that model. They include AOR2, 22,
25, 30, 38 and 44.
- Detailed case studies of BRM, often with the involvement of the authors, including
AOR22, 25, 28, 30, 42 and 87, together with ANR7 and 12. There is an overlap with the
previous category, on applications of the Cranfield approach.
- Surveys into the uptake of BRM, in different parts of the world, using questionnaires on
BRM practices. Usually the questions include asking about benefits measurement
practices. There are many articles of this type, including AOR8, 15, 16, 25, 27, 35, 54, 55,
58, 59, 64, 65, 81, 85, 89, 90 and 91 and ANR13.
- Surveys into BRM practices, specifically seeking to make the link with project
management or ICT investment success, including AOR9, 23, 63, 82 and 83, together
with ANR4 and 15.
- Research into the setting of targets in Business Plans, including AOR24, 34 and ANR20
- Roles played in BRM, including AOR10, 15, 33, 95 and 96, together with ANR21. In some
cases this covers responsibilities for measurement.
- Competences for BRM, including AOR4, 7, 18, 28, 62, 63, 71. Measuring benefits is one
of these competences.
- Typologies or critical analysis of benefits and associated measures, including AOR7, 11,
17, 36, 37, 47 and 87
- Papers on the assumptions underpinning BRM, including benefits definitions and
measurement, which include AOR20 and ANR8.
51
All these different categories of academic sources have something relevant to contribute on
benefits measurement. In some cases, this is because they have addressed measurement issues
directly, whereas in others there is a complementary focus to the research.
Table 4.2 Normative and descriptive publications
Normative/
Descriptive Count %
Descriptive 41 32%
Normative 34 27%
Descriptive and
Normative 17 13%
Descriptive leading to
Normative 28 22%
Not specified 7 6%
Total 127 100%
Table 4.3 Focus of guidance
Level Count %
Not specified 18 14%
Portfolio 6 5%
Programme 2 2%
Project 52 41%
Portfolio and Project 14 11%
Program and Project 20 16%
All 15 12%
Total 127 100%
52
and selected?
When does measurement occur?
There is a consensus that benefits should be identified at the start of a proposed change
initiative. In many instances the production of the Business Case is seen as the depository for
the information about predicted benefits (for example, AOR2, 24, 59, ANR19).
The Cranfield approach has as Step 1 ‘identifying and structuring the benefits’ but it is in Step
2, 'planning benefits realization' that the full business case is developed and measures for all
business benefits are included (AOR88, p69-74). 'Business benefits' are defined as 'an
advantage on behalf of a particular stakeholder or group of stakeholders' p70. The term
'outcome' is not used here, and indeed is not a term used extensively by Ward and Daniel.
Remenyi et al. (AOR74), similarly have a first stage when benefits are identified, followed by the
detail in the second stage . In 'initialisation of the project’, the overall objectives and expected
benefits will be identified, followed by the ‘production of the pictures’ – Business Picture,
Financial Picture and Project Picture - in which the benefits measures and targets will be
developed, mainly found in the Business Picture (p156-162 of the book). Benefits (point 1.10)
are derived from the outcome definition specified in point 1.5 of the Business Picture and the
chosen solution (1.7), so the terms outcome and benefit have a specific relationship to each
other.
Serra (AOR85) approaches benefits planning from an integrated portfolio/program/project

perspective, with business changes leading desired outcomes, leading to intermediate and end
benefits, which contribute to strategic objectives (Chapter 5) Thus benefits derive from
outcomes.
According to Zwikael and Smyrk (AOR97), fhe first stage in the life of a project is ‘start’ in which
the identification of target outcomes (generally up to 5 per project (p148)) is undertaken, which
should meet criteria of importance, measurability, lag and plausibility (p149). The business case
is judged according to the anticipated worth of a project, as flows of benefits, disbenefits, costs
and risk (p175). The relationship between outcomes and benefits is complex. 'Benefits are
driven by target outcomes. Benefits take the form of a flow of value from the perspective of the
funder to an entity arising from the generation of desirable outcomes. In general, the
relationship between outcomes and benefits is many to many' p51.
Therefore, four books produced by academics all have slightly different approaches to the
relationship between 'outcomes' and 'benefits'.
A number of academic sources provide empirical material on poor practice in the identification
of benefits. Amongst the findings are
53
 By and large, the need to articulate benefits, during a project’s planning phase, had
been recognized across projects, but all too often these benefits were either articulated
in a very general business sense, or in terms of the system’s functionality and features
or its intended usage, rather than clearly measurable business terms (Ashurst, Doherty
and Peppard p.363) (AOR6)
 In business cases and business briefs, benefits were identified 'but not in a coherent,
structured manner' (Baccarini and Bateup) (AOR8)
 Tendency for costs to be analysed in detail but "benefits are hardly measured"
(Berghout et al., Section 5.2) (AOR16)
Some questionnaire surveys provide quantitative data on organisational practices on benefits

identification, such as
 'The literature also prescribes that organisations establish a baseline prior to project
initiation to enable measurement of targeted benefits. On a 4-point Likert scale (from
“always to rarely”), only 30.8% of organisations always establish a baseline, followed by
34.6% of organisations reporting that they execute this most of the times. The remaining
organisations either establish a baseline some of the times (25.0%) or rarely (9.6%)'.
(Naidoo and Palk, p11701) (AOR65)
 The interviews were structured based on the Cranfield model. In the project proposal
section 24 out of 31 identified what measures are necessary to achieve the benefits but
only 11 of the proposals included information on how to measure the improvements
(Schwabe and Bänninger) (AOR81)
 In benefits identification, 72% assigned KPI’s or targets to the selected benefits and 41%
developed KPI’s for intangible benefits, such as staff morale (Smith et al. p1446)
(AOR85).
Many of the studies on benefits identification report on organisational practice in the 1990s or
early 2000s. In their comparative analysis of BRM practices in 2006 compared to 1996, Ward,
De Hertogh and Viane (AOR89) found that 'Overall, there is some evidence that over the past
decade organizations have somewhat improved on their efforts to identify and structure
benefits associated with their IS/IT investment projects. However, the numbers also reveal that
organizations still fail to take a full range of business benefits into account' (Section 5.3).
However, longitudinal studies are rare so there is limited evidence as to improvements in

practice over time.
In some cases, the identification of benefits is undertaken with the assistance of action
research. Caldeira et al. (AOR22) undertook a case analysis of an integrated paperless system in
a hospital in Portugal. This was a ‘before and after’ case study, with data being collected
between 2006 and 2011, covering periods before, during and after the implementation of the
paperless system, Alert®pfh. With the input of the researchers, '54 benefits were initially
identified, although some were subsequently discarded, grouped into 7 ‘macro-benefits. The
54
macro-benefits were represented as ‘benefits (main)’ in the BDN’ (Benefits Dependency
network) (p200) .
In another example, of a finance department in a local government city council in England,

business benefit measures were poorly defined, but were developed during the research, using
a participative approach (Coombs) (AOR25).
Another feature of benefits identification is the danger of optimism bias affecting the process
or even benefits distortion. Prater, Kirytopoulos & Ma (2017) (ANR20) provide a quantitative
literature review of optimism bias in projects, and found 33 sources on this topic. Fearon and
Philip (AOR36) outline how in projects in the insurance industry expectations about benefits are
subject to external influences including software vendors, technologists and the partner with
most power, prior to project sign-off. Love and Irani (AOR56) explored the benefits
management practices of SME's in the construction industry and found that investment
justification may be the result of "assigning arbitrary values to benefits and costs" (p228), and
be a form of "creative accounting". Dupont et al. (AOR33) found that theoretical benefits are
defined by senior management but are subject to distortion if implementation is not achieved
and compliance is low.
Focus on outcome and/or intermediate benefits?
As referred to above, academic books covered in this review draw a distinction between
outcomes and benefits. No instances were found of the use of the term 'outcome benefit'.
There were, however, sources which used the following terms
- intermediate/end benefits (Serra/Serra and Kunc) (AOR82-84)
- leading/lagging benefits (Mossalam and Arafa) (AOR64)
- short and long term benefits (Paivarinta et al.) (AOR71)
- means and ends network (Barclay and Osei-Bryson) (AOR14)
Chih and Zwikael (AOR24) review the question of comprehensiveness in relation to types and
levels of benefit. They conclude that, 'Unfortunately, there is no universal answer as to what
can be considered “comprehensive,” because it varies from one case to another. As a guideline,
Henderson and Ruikar (2010) suggested different categories of target benefits including
direct/indirect, short/long term, internal/inter-organizational and economical/cultural benefits’
(p358).
Any kind of sub-division of benefits will be reflected in benefits dependencies mapping. The
Cranfield approach uses the benefits dependency network (BDN) which includes just one
column for benefits, entitled ‘Business Benefits’ (Ward and Daniel, p96) (AOR88). This basic
model is used by many of the sources in the academic review, and has been subject to
amendment and refinement. Coombs (AOR25) adds inhibitors into the structure of the BDN in
55
between ‘Business Changes’ and ‘Business Benefits’ to reflect the difficulties in his case study
organization in achieving change, but retains one category of benefits. However, it should also
be noted that although the Cranfield BDN is one of the simpler benefits mapping techniques
available, complaints about it are sometimes made that it is too complex. Eckartz et al. (AOR34)
found that, ‘A more straightforward connection between the benefits, goals and drivers of the
project is demanded’ p4647. Their proposed improved approach "assigns a sequence in
achieving the benefits" p4649, perhaps reflecting a need to establish interdependencies
between benefits.
Other versions of benefits mapping incorporate different levels of benefit explicitly. For
example, Serra (AOR84) links intermediate and end benefits in his version of the BDN, but also
mentions that ‘there are cases where some steps, such as business changes and intermediate
benefits will not be needed. In these cases, such steps can be skipped’ p101.
In some instances, interdependencies are related to categorisations of benefit from other

models. For example, Liles (AOR52) using the Balanced Scorecard as an approach for mapping
benefits, established a framework whereby organisational learning benefits lead to customer
satisfaction, leading to financial benefits.
Linking to specific stakeholders
Many of the sources stress the importance of stakeholder engagement in the process of
identifying benefits, and specify key categories of stakeholder. Chih and Zwikael (AOR24) state
that
The first critical factor suggested by our participants is stakeholder engagement in

formulating target benefits, which is in line with Breese (2012). Public project
stakeholders who need to be engaged in target benefit formulation may include
governing stakeholders (e.g., senior executives), supporting stakeholders (e.g., IT
departments) and end users. It is essential to engage the “Right stakeholder for the right
reason at the right time” P358.
In many cases, organizations fail to recognize the stakeholder-specific nature of benefits.

Coombs (AOR25) remarked that
More specific advantages that could be associated with a particular stakeholder or

group of stakeholders were categorized as business benefits e.g. improving the speed
and accuracy of month-end and year-end reporting. However, although these
advantages were labeled as benefits in the documentation from the case study site, the
explicit association with particular stakeholders was rarely stated. P369.
Many sources (eg Laursen and Svejvig (AOR50)) highlight that fact that different stakeholders
will value benefits in different ways, and link this with the power exerted by dominant
stakeholders to determine which benefits are prioritized (eg Breese (AOR20)). Ward and Daniel
56
(AOR88) incorporate stakeholder analysis into the Cranfield method, providing frameworks for
categorizing them according to power and influence, balance of benefits/disbenefits and overall
position in relation to the change being proposed. However, applications of the Cranfield
method do not always incorporate the high emphasis on stakeholder management. For
example, Flak et al. (AOR37) suggested that the Norwegian approach to BRM seems to provide
less explicit focus on stakeholder involvement compared to the Cranfield Process Model, p10.
Examples of benefits of particular types (intermediate or outcome) being linked to specific

stakeholders are not evident. An emphasis on end user benefits is stressed in some
publications, for example, by Doherty (AOR31), who links BRM with the socio-technical studies
literature and highlights the importance of end user benefits based on socio-technical
principles. However, benefits for end users might be ‘intermediate benefits’, with the outcome
benefits defined as being higher productivity for the organization as a whole, or they might be
classed as ‘outcome benefits’ if intermediate benefits are linked to greater functionality, for
example.
For Zwikael and Smyrk (AOR97) benefits are defined in relation to the interests of the funder
(P276). The question as to whether user or funder stakeholder interests have greater influence
over benefits measures was discussed by Breese (AOR20) for regeneration programmes in the
UK, where government as the funder set the benefit definitions and benefits management
frameworks, rather than local communities.
Micro-scale studies of stakeholder-related benefits are scarce. However, one detailed research
study (Nogeste, ANR17) linked specific benefits to individual stakeholders in five case studies of
government projects and programmes in Australia. The purpose of the research study was to
develop a method for improving the definition and alignment of intangible outcomes and
tangible outputs. This study explored benefits as intangible outcomes from the perspective of
21 different individuals involved in these projects and programmes.
Who assesses the benefits?
In the academic sources there is a lack of attention to the detail of measurement, but more
interest in overall accountabilities for benefits. Many studies identified a lack of ownership, and
57
if this is the case it is unlikely that anyone will be assessing benefits on a rigorous basis.
Examples of this lack of ownership include
- Moreover, there was absolutely no evidence of organizations explicitly identifying

owners for these benefits, to help facilitate their ultimate realization. The difficulty of
getting organizations to provide clear measures for benefits was highlighted by a project
manager {P21} who lamented: ‘At the start of the project we asked about success
criteria and how they (the customer) would measure return on investment. All we could
get out of them was that other players in the market already had similar technologies,
and they wanted to eliminate all paper from their sales cycle’. (Ashurst et al. p 363,
AOR6)
- A research team developed a Benefits Realisation Management Process (BRMP) for
Manchester, Salford and Trafford (MaST) Local Improvement Finance Trust (LIFT) to fill a
gap in responsibilities for assessing benefits for the project (Harris et al., AOR43).
A variant on lack of ownership is where responsibilities were clear during project

implementation, but less so after handover to the business, when benefits should be realised.
- an overwhelming majority of respondents (51.9%) revealed that either the project

sponsor or project manager (25.00%) was ultimately accountable for delivering on the
expected benefits……. only a minority of organizations (19.2%) reported that the
Business Unit Head who is ultimately the recipient of the benefits was accountable for
delivering on the expected benefits (Naidoo and Palk, p11699-11700, AOR65)
- assigning explicit accountability to business managers for realizing specific benefits is
performed by only 36% of the respondents. Over the past ten years, this crucial practice
shows only a minimal improvement of 4% (Ward, De Hertogh and Viane, Section 5.5,
AOR89).
One of the few studies which linked the realisation of benefits back to measures was Waring et
al. (AOR91). In their study of benefits realisation in acute hospitals in England, the response to
the question ‘Staff within my area of responsibility are able to realise benefits from IT projects
through the use of metrics to measure success’ was
18% strongly agree
51% agree
18% neutral
13% disagree
0% strongly disagree (p14)
58
While these results suggest a higher degree of commitment to benefits realization than other
studies, it is not clear exactly who is assessing the benefits, in the wording of the question
posed. As far as the guidance is concerned, this should be vested in a benefits owner. According
to Ward and Daniel (AOR88), ‘the benefit owner should ‘be an individual who gains the
advantage inherent in the stated benefit and therefore is willing to work with the project
team......to ensure that the benefit is realised’ p103. A similar approach is advocated by Serra
(AOR84) with benefit owners taking responsibility for benefits realisation (p80-81) Zwikael and
Smyrk (AOR98) use a different term, but specify a similar role. They argue that a ‘project owner’
should take responsibility for benefits realization, as the funder is too senior and busy and the
project manager is preoccupied with delivery. The project owner should be a senior executive
who might be responsible after project completion for any ongoing operation….(p855).
Cha (ANR4) focused on the role of the ‘project owner’- from the client side perspective, where
IS projects are delivered by suppliers and then handed over to the client. He explored 31 IS
project cases in the UK public sector and adopted content analysis (mix of quantitative and
qualitative analysis) by examining National Audit Office reports. He identified dymanic
capabilities required for the role, with training and skills development and knowledge and
experience transfer being key back-end skills, to accelerate post-implementation benefits after
a project is completed.
Whether the ‘project owner’ is the same person as the ‘project sponsor’ will vary according to
organisational circumstances. Whether they are the same person or not, the understanding of
the project sponsor of their role in relation to benefits realization will have a material effect on
whether those benefits are attained. Recent research by Turner (AOR28) in an acute hospital in
the UK suggests that project sponsors may not necessarily see their role as being concerned
with benefits. The research explored how the project sponsor role is experienced and what is
understood by the senior managers who undertake the role, and, what if anything do they
understand of benefits realization. The study identified 3 conceptions of the project sponsor
role. In ‘just doing the day job’ no awareness of a role in realizing benefits was experienced. In
‘the capable manager’ benefits are identified as part of delivering projects. Only in the ‘wearing
two different hats’ role is there an understanding of a responsibility for realizing benefits.
While the general view is that benefits owners should be assigned to be responsible for benefits
realisation, Pedersen (ANR18) presents a contrary view, that ‘In the literature, governance
issues are primarily dealt with by insisting on appointing benefits owners (Ward and
Daniel,2006).The literature does not reflect the organizational and technical complexity involved
in this case’ (of government initiatives in Denmark) (p281). This is because of the complex inter-
relationships between different levels of government for local roll-outs of national initiatives,
which mean that so many stakeholders are involved and responsibilities have to be shared.
59
The academic sources provide a variety of insights on this issue from different perspectives –
descriptions of existing practice, normative guidance and a mix of the two.
In many cases emergent benefits were not mentioned (66% of sources) implying that the issue
was not considered in the research. In most cases where it was considered, either implicitly or
explicitly, the importance of an emergent approach was acknowledged. For example, Ashurst,
Crowley and Thornley (ANR2) claim that ‘there is a strong emergent element to any significant
change programme and it is foolish to assume that all benefits can be identified in advance’
p14. Kagioglou and Tzortzopoulos (ANR8) suggest that ‘Implicitly, measures will change, and
NPD (New Product Development)and BRM should be designed not only to incorporate emerging
changes but to encourage ‘fixed solutions’ to be opened up and revalidated to embed a culture
of continuous improvement P185.
The significance of emergent benefits has long been recognised, but often organisations fail to
incorporate appropriate processes for capturing them. Ward, Taylor and Bond (AOR90)
identified as long ago as 1996 that ‘The majority of respondents, 86% (51/59), believed that it is
not possible to anticipate all potential benefits at the project approval stage. However, only 19%
(11/59) of respondents claimed to have a process for taking advantage of this fact in order to
identify further benefits after implementation, and take action to realise these benefits. The
implication is that there are often more benefits to be gained after implementation, evaluation
and realisation of IS/IT benefits but that current practices mitigate against exploring these
potential further benefits’ p223.
Smith et al. (AOR85) identified that ‘74% of participants’ organisations do not have a formal
process in place that identifies any further benefits after implementation. Of the 26% that had
such a process in place, all of them indicated that their organisations take action after
implementation to realise these additional benefits’ P1450.
Lin et al. (AOR55) found that 52.4% of the Taiwanese SME’s in their survey claimed to have a
process for identifying further benefits after implementation, which was higher than the results
obtained by Ward, Taylor and Bond (AOR90 - see above) and by Lin and Pervan (AOR54).
The Cranfield method includes as the final Step 5 ‘Establishing the potential for further benefits’
(Ward and Daniel, p79) (AOR88). When organisations adopt that method, this provides a basis
for the addition of benefits measures over the life of the project (eg Divendal’s research with
Heineken in the Netherlands (AOR30)). Serra (AOR84) includes guidance on ‘identifying
potential for further benefits’ as part of ‘reviewing and evaluating benefits’, p163, while Zwikael
and Smyrk (AOR97) use the term ‘fortuitous benefits’. The BeReal method is concerned with
both planned and unplanned benefits, with emergent benefits being part of Phase 3 (Harris et
al. (AOR43), Sapountzis et al. (AOR80), Yates et al. (AOR95).
60
However, there might be reasons in some organisations to concentrate on the original set of
benefits measures in terms of organisational capacity. For example, Marshall and McKay
(AOR59) reported that ‘Interestingly, our CIOs tended to adopt the satisficing position (i.e. that
delivering, say 80% of expected benefits was probably good enough and that the resources
consumed in trying to achieve 100% or more would be better diverted elsewhere’ p24. Another
potential issue with emergent benefits is that it might divert away from the rigour of the initial
benefits identification process. Thus Marnewick (AOR58) emphasises definition of benefits and
their measurement before project implementation rather than emergent benefits: ‘It is
important that promised benefits must be properly formulated at the beginning of the project’
p757. There may be cases where the initial emphasis is on the original benefit targets, and once
these are achieved a more emergent approach can be adopted. Badewi et al. (ANR3) suggest
that the project starts by targeting core benefits, and once these benefits are realized attention
can turn to investing to realise innovative benefits.
42% of sources indicated that benefits measurement needed to continue post-implementation,

either on an on-going basis or over a specific timescale (Table 4.4).
In a few cases, research has specifically asked about the extent of post-project measurement of
benefits. Waring et al. (AOR91) asked their respondents in acute hospitals in England if ‘Benefits
realisation continues to be monitored up to one year after an IT project is completed’ and
obtained the following results,
9% strongly agree
25% agree
31% neutral
30% disagree
5% strongly disagree p11.
Schwabe and Bänninger (AOR81) found that ‘The majority of all companies (58%) revisit the
benefits once at the project end and then stop caring about benefits. 19% trace benefits until it
becomes clear how they will influence the next budget, i.e., at the most one year; the remaining
23% trace benefits up to three or five years (as it is required by their investment calculation)’ p5.
61
Table 4.4 References to ‘close out’ timescale
Close Out Count %
NS 75 59%
Ongoing 44 35%
3 years 2 2%
1 year 1 1%
6 months 4 3%
1 review 1 1%
Total 127 100%
Key
NS: Not specified – The paper has not covered this point
Ongoing: the publication addresses the importance of a continuous review until the maturity of
benefits achieved or to be fully integrated to the KPIs of the organisation (transferred to other
departments)
3 years, 1 year, and 6 months: the review is continuous or happen in intervals but ends after the
time indicated
1 review: the review happens only one time after the transition process is completed.
Measurement post-implementation faces the hurdle that the project team may no longer be in
a position to contribute to the benefits realization process. Young and Jordon (AOR95)
identified that ‘It takes a relatively long time to realise benefits from an IS investment thus the
majority of the benefits of an IS investment are realised after a project team has disbanded’
p714. This means that the resourcing of benefits measurement may be problematic. Marshall
and McKay (AOR59 noticed a tension between the resources required to do a full audit of the
benefits that were promised initially and the idea of satisficing and recognising new
opportunities.
62
Measurement post-implementation can have different purposes. Lin and Pervan (AOR54) found
that 77% of the large Australian organizations they surveyed conducted post-implementation
reviews, but this was mainly concerned with lessons learned for future reference, rather than
to identify and realize further benefits. In contrast, Lin et al. (AOR55) found in their survey of
Taiwanese SMEs that more organizations claimed to have a formal process for identifying and
realizing further benefits (52%) than had a formal process to ensure that lessons were learned
(48%) p53.
Post-implementation measurement by projects and programs may be undertaken within the

portfolio review process. Ward and Daniel (AOR88) recommend that at portfolio level an
annual review of investments completed in previous year and the cumulative benefits delivered
should be undertaken (P224). Mihic et al. (AOR61) identified in their research into energy
efficiency projects in Serbian public buildings that ‘Evaluation results are collected in a special
evaluation report, which serves as a platform of providing post-project benefits. Evaluation
reports of individual projects are collected into a unique evaluation report of the entire portfolio’
p71.
the customer is?
Tables 4.5 and 4.6 summarise the publications in the academic literature according to the
nature of the investment and sector. Thus, for example, there are descriptive case studies
which are about the benefits from IT investments in hospitals. There are books which are
concerned with IT investments but apply across all sectors, so are classed as ‘general’ in Table
4.6.
Table 4.5 Publications by nature of investment in the academic literature review
Nature of
investment Count %
IT 86 68%
Construction 7 6%
Others 20 16%
Theoretical 14 11%
Total 127 100%
63
Table 4.6 Publications by sector in the academic literature review
Sector Count %
Private Sector 1 1%
Public Sector 16 13%
Manufacturing 6 5%
Financial Services 9 7%
Hospitals 8 6%
ERP Systems/IT 8 6%
Other/General 79 62%
Total 127 100%
Tables 4.5 and 4.6 show that the literature is mainly concerned with IT investments, and that
the sectors involved are diverse. The focus on IT investments helps to explain the many
examples of sources which use the Cranfield method (Ward and Daniel (AOR88)) as the basis
for analysis or as a model for adoption in practice. The work of Ashurst (AOR4-7 and ANR2) has
also been used as a theoretical framework, where the focus is on competences (see Pedersen’s
critical analysis of government projects and programs in Denmark (ANR18)). The theoretical
contributions of Ward and his colleagues at Cranfield and Ashurst have been orientated
towards counterbalancing the tendency to focus on the technology, rather than seeing
information technology as a means to an end to achieve IT-enabled business improvement.
The different investment type and sectoral categories are discussed in Question 5
64
Table 4.7 Incidence of types of benefit measure
Types of benefits
measure Count Ratio
Not Specified 36 28%
All measures used 33 26%
One or two types of

measure used 44 35%
Other answers 14 11%
Total 127 100%
As can be seen from Table 4.7, 72% of the publications addressed the issue of types of benefit
measure, with 26% of those recording the use of all three types.
From Question 4 it is apparent that the academic literature is dominated by IT investments, and
so many of the publications address the benefits measures used to assess IT-enabled change.
Remenyi et al. (AOR75) is a key source, because it incorporates an investment appraisal
perspective to provide detailed and comprehensive guidance on the measurement of benefits
from IT investments and their management, including costs as the other aspect of the value
equation. Amongst the relevant material to be found in this publication is
 A summary of the recent history of the development of ICT evaluation (Chapter 1).
Covers different methods in detail.
 A taxonomy of techniques comprising Fundamental methods, composite methods
(usually used in ICT evaluation) and meta approaches (p48+
 Description of twelve different methodologies available to assess the performance of IT,
which 1. Strategic match analysis and evaluation 2. Value chain assessment
(organization and industry) 3. Relative competitive performance 4. Proportion of
management vision achieved 5. Work study assessment 6. Economic assessment – I/O
analysis 7. Financial cost benefit analysis 8. User attitudes 9. User utility assessment 10.
Value added analysis 11. Return on management 12. Multi-objective, multi-criteria
methods, p111.
65
 A list of factors for an IS selection system covering strategic value, critical value,
operational value, architectural value, investment value and risk assessment, is
provided, to use in a ranking and scoring method (Chapter 11).
Remenyi et al. (AOR75) also list four key problems with ICT benefit measurement and
management, which are
 benefits and identifiable performance improvements

 information systems reach,
 tangible and intangible benefits, and
 evolution of benefits, (p27-30)
Under ‘Benefits and identifiable performance improvements’ Remenyi et al. suggest that no
special ICT benefits metrics need to be developed; it is best to use general business
performance indicators.
Many of the difficulties in ICT benefits management are highlighted in the descriptive sources in
the academic sources review. Coombs (AOR25) refers to the implications for the use of
appraisal techniques of the ‘significant gap between costs being incurred and the actual
realization of benefits’, P364. Doherty et al. (AOR32) claim that
Unfortunately, there is very significant gap between simply specifying the desired
outcomes of a prospective software development project, in financial terms, and
ultimately establishing the veracity of such cost savings or improvements to revenue,
once the system is operational. This is partially because it is far less easy to effectively
measure the outcomes of a system’s project, in financial terms, than it is to make pre-
investment predictions. However, probably, the more significant reason that anticipated
benefits, whether financial or otherwise, rarely translate into actual benefits is that
project teams typically fail to recognise the critical role of organisational change.
Ashurst (AOR4) refer to pressures to force the non-financial benefits of strategic innovation
projects to be translated into financial impacts (p67). The consequence is that such projects
lose out to less innovative IT projects where a financial case is easier to construct (p68).
Apart from ICT, the other investment type represented in the academic literature is
construction. Baccarini and Bateup (AOR8) found that office fit-out projects used
 Efficiency benefits measured by increase in occupancy density. Reduction in floor space

per person.
 Collocation benefits; occupant productivity; flexibility for business growth; image
benefits; desired corporate look; impact on environment; impact on behaviour e.g.
collaboration,staff retention.
However, KPIs were only set for efficiency measures.
66
In another construction example, Love et al. (AOR57) developed a BRM building information
modelling framework, and identified a different list of potential benefits - Facilities
Management labour utilisation; time to close out a works order; utility cost reduction and
energy savings; material wastage; fuel savings (transport); regulation compliance; improved
inventory; configuration management.
In general, there is not enough information on different sectors to do more than identify any
individual case studies which provide good examples of the types of benefits for that sector. For
example Caldeira et al. (AOR22) provide a case study of a paperless system introduced into an
acute hospital in Portugal. They found that
‘Some of these benefits are immediately apparent from a financial perspective. Other
benefits do not show a clear financial value, and have been classed as measurable,
quantifiable or observable in accordance with the methodology adopted. There are still
others that are potentially financial, but it is not yet possible to identify or estimate their
monetary value accurately. Nonetheless, the benefits deriving from implementing ALERT
makes it possible to estimate an annual reduction in costs or gains achieved of 3,869,896
euros’ P199-201.
‘The macro-benefits include greater precision in diagnoses and clinical prescriptions;

reduction in costs for tests and clinical analyses; greater systematicity in information for
management purposes; reduced personnel costs; reductions in costs for facilities,
equipment and material supplies; improved service for patients; improved working
conditions for health workers; and the capacity to increase the volume of activity,
especially in outpatient appointments, with no extra expenditure’. (P202)
Some sources integrate BRM with other methods, such as the Balanced Scorecard (Gomes and
Romao (AOR38) and Steinfort (ANR26)) Some BRM approaches incorporate complementary
participative approaches for deriving measures for quantitative and qualitative data (Harris et
al. (AOR43)). However, participative approaches do not always go according to plan. For
example, Divendal (AOR30) noted that
‘Adding benefit details in the benefit templates is an easy task for the participants, until
the benefits need to be quantified. Quantification is a difficult task and after more than
two hours in the workshop, only some ideas on how one could quantify every benefit are
provided. It becomes obvious that identification of benefits and quantification of benefits
could better be split into two separate workshops for complex projects’p44.
Breese (AOR20) refers to quantification as one of the characteristics of the ‘modern paradigm
of management science’ which underpins BRM but whose assumptions do not reflect the
nature of human experience, p342. Relating this issue to survey findings from the Swiss
financial sector, Schwabe and Bänninger (AOR80) found that ‘two thirds of the companies (21)
regarded qualitative benefits equally important as quantitative benefits, in the majority because
67
strategic benefits are typically strategic (13). One third (10) regarded quantitative benefits as
more important, because qualitative benefits are difficult to capture (5) and because, in the end,
only money counts (5)’. P3
implementation?
The research by Cha (ANR4) and Turner (ANR28) referred to in Question 3 is particularly
relevant to this issue, focussing on different aspects of the role of ‘project owners’ and ‘project
sponsors’ respectively. Identifying the kind of behaviours and aptitudes at senior level which
can stop this gap from opening up is a key challenge.
68
5. Reflections
For many years, there has been guidance on BRM from the work of the pioneers (authors such
as Bradley, Remenyi, Thorp, Ward and Daniel) which provides methods for incorporating
measures into the BRM life-cycle. At the same time, many descriptive studies have identified
the shortcomings in practice over the years, both in measurement for forecasting purposes and
in measuring benefits realisation, for which there are many reasons.
One of the factors is the difficulties in incorporating BRM into project, program and portfolio
management. Much of the guidance on P3M misses out key aspects of good practice in
measuring benefits, evidenced by the many sources which did not address the questions in our
review (although this is partly due to the inclusion of publications which are not primarily about
BRM). Furthermore, there is no consensus on how benefits fit into P3M, and it is only recently
that guidance on the three levels has become more coordinated, with umbrella sources on
governance between them.
Another reason for poor practice is that measurement raises difficult issues, highlighted by
Remenyi et al (AOR75), and explored in this report, of
- benefits and identifiable performance improvements (including attribution)
- information systems reach,
- tangible and intangible benefits, and
- evolution of benefits, (p27-30).
and selected?
There is a consensus across the different literatures that outcome benefits measures need to be
identified early on in the inception of proposals for investments in change. However, there are
some differences in the emphasis in different types of guidance. For example, the Cost Benefit
Analyis/Business Case guidance tends to focus on ‘before and after’ measurement while the
P3M BRM guidance tends to see measurement occurring throughout the life cycle or at key
stages (e.g. end of tranches in MSP).
This study has encountered difficulties with terminology. Our first study identified the lack of
consensus on the definitions of terms such as ‘benefit’ and ‘benefit realisation’, and
inconsistencies in that terms such as ‘benefit’ and ‘value’ are sometimes viewed as synonymous
and sometimes given distinct meanings. This study has highlighted the relationship between
69
the terms ‘outcome’ and ‘benefit’ as being similarly inconclusive. The different relationships
between them include
- benefits are considered to be the outcomes favourable to an organisation, so the terms
are virtually synonymous
- benefits follow on after, or result from, outcomes so they are separated in terms of
stages in a process
- benefits are the measures of the outcome
- one or the other term being used, but not both.
There may be a one-to-one relationship between outcomes and benefits, or there may be more
complex inter-relationships between them. Added to this, are many other terms, such as KPI’s,
success factors, results, outputs and deliverables to be added into the P3M terminology used in
organisations.
The terminological issues continue into sub-divisions of benefits. There are a number of
dualities (paired terms) found in the literature, including
- outcome-intermediate
- end-intermediate
- ultimate-intermediate
- lag-lead
- long term-short term
There are different ways in which distinctions are made between the paired terms, including
- By importance, in the eyes of beneficiaries or other stakeholders
- By relationship to strategic objectives
- By time of occurrence as to when they happen
- By time as to when they can be measured to monitor progress
- By degree of attribution to project and program activity.
Added to the diversity in the relationships between terms are the variations in the links
between benefits, measures and metrics. Many sources use either the term measure or metric,
and would consider them as synonyms, with a one-to-one link with a benefit, but in some cases
a less simple approach is adopted. Examples are the different publications authored by or
influenced by Gerald Bradley, for whom more than one metric may be needed for a measure,
and more than one measure may contribute to the tracking of a particular benefit.
Many of the variations in terminology are of a semantic nature and do not matter too much,
but unless guidance is expressed clearly and unambiguously with the relationship between
different terms being explained it can contribute to confusion in the practice of BRM, in view of
the preference for methods which are simple and easy to understand and use (Paivarinta et al.
(AOR71))
An example of this is guidance on benefits dependencies/benefits mapping. With clarity on the

use of terms and the causal relationships between different terms used in them, benefits maps
70
can be an effective tool for establishing strategic alignment and provide a framework for
benefits tracking and realization.
An area where the guidance is sometimes clear but in other cases patchy is in the position of
benefits in relation to stakeholders. Some methods, such as Social Return on Investment (SROI),
derive benefits from stakeholder involvement, and inclusion of the user perspective, and also
the role of other key stakeholders, such as funders, should be a ubiquitous feature of guidance
on BRM. However, many sources neglect this topic or refer vaguely to the importance of
stakeholders, without acknowledging that benefits have differential effects on different
stakeholders and will hence be perceived in different ways.
There are a large number of different roles described related to benefits measurement, and they imply
that in most large organisations a variety of different individuals will contribute. This will include
strategic roles with oversight of processes, middle level roles for benefits coordination and
interpretation of data and lower level roles for the collection and analysis of data. How the management
of benefits relates to the management of P3M will vary between organisations. Many sources of
guidance are prescriptive about roles, emphasising the key position of the Business Change Manager or
Benefit Owner, on the basis that this provides a model which individual organisations can follow if they
want, or can adapt to their own circumstances.
Recognising that different stakeholders need to be involved alongside clarity of roles, some sources of
guidance advise on the use of RACI charts to define roles.
While there is a consensus that benefits need consideration from the inception of any proposals for
change initiatives, there is less agreement on the stage at which benefits measures should be identified
and defined for monitoring purposes. Some sources of guidance suggest that this happens as a gradual
process, with benefits first being identified as qualitative concepts and later being turned into measures
for tracking.
While guidance on business cases/cost benefit analysis often focuses on planned/forecast benefits, most
sources of P3M guidance recognise emergent/unplanned benefits. However, a key issue is how far the
emergent approach to change should be emphasised in comparison to the monitoring and management
71
of benefits in the business case. To what degree should benefits measures be added and existing ones
adapted. This is one area where in-depth studies of what happens in practice over the life-cycle of a
project are required.
This issue links to the question as to when benefits are realised, and assumptions over the attainment of
benefits at the project, program and portfolio levels. It also relates to the nature of the change initiative,
and whether there is a distinct separation between implementation and use. As with emergent benefits,
there is some evidence of an increasing recognition of the significance of this issue, for example, in end
of tranche and post-implementation reviews. Unless the handover stage to BAU involves a commitment
to benefits measurement and management, overcoming resistance to on-going measurement, benefits
are unlikely to be optimised and sustained. Long term commitment reflects whether the business truly
owns the initiative.
the customer is?
Most of the guidance is cross sector, although many sources are aimed at the public sector
generally or Government Departments specifically. In industry terms, ICT is far and away the
most frequent sector, with bespoke guidance and many more descriptive studies than any
other sector. Combining the public sector and ICT, there is specific guidance for, and relatively
numerous research studies on, e-Government.
There too few empirical studies to make contrasts on the basis of project risk or project type.
One way in which the guidance can be distinguished is in the breadth of the target audience.
Historically, the literature on ICT has been aimed at professionals beyond P3M, to achieve the
change in mindset from ICT investment to ICT enabled business improvements. In addition,
some recent research and guidance, particularly from the PMI, explicitly aims at a wider target
audience than the project management profession, because of the wider significance of BRM.
There is no shortage of different categorisations of measures and advice on the relationship between
different types of measure. However, measurement can be difficult, perhaps particularly for more
mature organisations who have the processes and procedures for BRM in place and therefore focus on
how to make them work (see the PMI Thought Leadership Series- Strengthening benefits awareness in
the C-suite report (PBR19)).
For ICT there is a wealth of information available on methods and tools for measuring benefits (see, for
72
example, Remenyi et al. ‘The effective measurement and management of ICT costs and benefits
(AOR75)), but many of these methods are generic and can be used for any type of investment. Similarly,
development of directories of measures as applied in different industries can help to build up the
evidence base on measurement in different contexts.
Some sources make suggestions as to the number of benefits, or the proportion of possible benefits
which should be tracked. However, there is a trade-off between maximizing the number of measures
tracked to ensure all relevant benefits are captured and minimizing the resources which need to be
allocated to benefits measurement and monitoring. This is an issue which has to be addressed on a
context-specific basis.
Another issue of contention is whether a benefit has to be measurable to be taken into account (Ward
and Daniel (AOR88) include a category of ‘observable’ but not ‘measurable’). The use of proxy measures
means that virtually anything can be translated into an indicator which is measurable, but that indicator
might not be a true embodiment of the nature of the benefit. A further area of contention surrounds
the monetisation of non-financial benefits. While there are a number of consistent methods of doing
this, the attempt to give a financial value requires assumptions being made which may not prove valid.
On such issues, organisational culture and context may put pressure on those measuring benefits to
adopt ‘objective’ measures, even though the act of creating those measures may, in fact, involve a high
degree of subjectivity.
implementation?
This issue is one which can be addressed by bringing together evidence outlined previously, in the other
questions. Despite the wealth of guidance on benefits measurement, descriptive studies of current
practice often report highly inaccurate forecasting of benefits in the business case (see Prater et al.
(ANR20), for a systematic review of optimism bias); and organisations failing to measure the actual
realisation of benefits (see the UK National Audit Office reports in the Government Bodies Review). The
lack of accountability for benefits leads to project implementation proceeding without a commitment to
benefits realization (see, for example, Smith et al (AOR 85) for a description of survey results
exemplifying this problem).
As this issue becomes increasingly recognized, ways to address it are being formulated, for example in
the increasing attention paid to handover to BAU, the use of RACI frameworks to recognize the plurality
of roles involved in BRM and in research projects on the roles of those who should be accountable (see,
for example two doctoral theses, by Cha (ANR4) and Turner (ANR28)). However, the underlying issue is
whether the organization truly owns the initiative.
73
Other disciplines and literatures relevant to the research
As part of the study, the research team were asked to briefly refer to bodies of literature outside the
BRM/Project management literatures which are relevant to the topic area of benefits measures,
including the possibility of more evidence to enhance the analysis. Some of the sources we have covered
incorporate insights from other disciplines, and the boundaries between management topics are
inherently fuzzy. Four areas are covered below,
 Change management
 Investment appraisal
 Evaluation
 Performance measurement/performance management
Change management
While the Professional Bodies review includes some publications on change, the explicit incorporation of
BRM in the literature on change management is rare. None of the academic sources come from journals
on change, such as the Journal of Organizational Change Management and the Journal of Change
Management. The development of theory on emergent change would provide additional backing for the
dynamic nature of benefits measurement, linking it with propositions and evidence on the
unpredictability of the business environment.
Investment Appraisal
While the development of BRM arose partly from dissatisfaction with investment appraisal techniques
in the 1980’s, particularly for IT initiatives, the development of BRM and particularly benefit
measurement draws heavily on investment appraisal methods, and the boundaries between BRM and
investment appraisal are fuzzy. Some empirical research studies have asked whether organisations used
Investment Evaluation methods and/or BRM (for example, Lin et al. (AOR55), but most studies have
referred to techniques such as ROI and CBA as being part of benefits measurement. The report has
drawn on ‘The effective measurement of ICT costs and benefits’ by Remenyi et al. (AOR75) but there will
be other sources from this literature which would provide complementary insights.
Evaluation
The boundaries between investment appraisal and evaluation are fuzzy, and many of the sources drawn
upon for benefits measurement use the term evaluation to describe their activity, for example, the SROI
sources drawn upon in this report. Some of the theories used in the evaluation literature, have their
parallels in BRM methods, such as ‘theories of change’, which have common elements with the theory
behind benefits dependencies/benefits mapping. ‘Evaluation’ is an example of a journal which is not
drawn upon in this study, which would provide insights relevant to benefits measurement.
74
Performance Measurement/Performance Management
Performance measurement/performance management is linked to the other three literatures referred

to above and is also drawn upon extensively in many of the BRM sources. For example, the Balanced
Scorecard of Performance Measures is a tool which is often used as part of BRM methods and research
(See, for example AOR38 and ANR26 in the academic sources review). Further use of this discipline
would be useful in exploring issues such as the distorting effects of targets upon behaviour.
75
6. Conclusion and Recommendations
The study has identified a ‘knowing-doing’ gap in benefits measurement, whereby there is a
gap between what should be and what is, in terms of ‘state of the art’ as described in the
guidance, and the ‘state of the art’ as evidenced in current practice. Guidance is accessible but
there is less evidence easily available describing existing practice, because much of that
evidence is in academic sources. Therefore,
Recommendation 1 is that there should be more initiatives to make the evidence on practice
on benefits measurement and management more accessible to practitioners and other
interested parties.
There are terminological inconsistencies affecting benefits measurement, for example, in the
different relationships between the terms ‘outcome’ and ‘benefit’ in different guidance
sources. There are also a variety of different dualities (paired terms) used to categorise
outcomes and/or benefits, such as intermediate/end, leading/lagging and short term/long
term. The roles of projects, programs and portfolios in benefits realisation translate across into
benefits measurement, so clear terminology is required to enable clear and consistent
messages to be provided in guidance. This will help with clarity on what needs to be measured
and how it is to be measured (in conjunction with Recommendations 3 and 5 below, on
ownership of the process). Therefore,
Recommendation 2 is that consistent and clear sets of terms be developed for guidance on
the causal relationships involved in strategic alignment of project activity, incorporating
program and portfolio levels where organisations utilise these levels, which will in turn
facilitate the benefits measurement process. This links especially directly into the use of
benefits dependencies/mapping tools.
P3M does not exist in isolation - it exists within the broader context of overall enterprise
governance and management - from strategy through to operations. The diverse roles of
different stakeholders in all aspects of P3M are increasingly recognised, but the stakeholder-
specific implications of benefits targets are not always appreciated. There is a need for clear
ownership of the BRM process and for the owners to involve key stakeholders in identifying
benefits and developing benefits measures and targets, and then also keep them engaged
during the benefits life-cycle, so the whole organisation is committed to the optimisation of
benefits. Key stakeholder groups include users and may include funders, but could also cover
any individual or group with high power and high interest. Therefore,
Recommendation 3 is that guidance needs to build on progress towards owners of the BRM
process engaging key stakeholders in the development of benefits measures and
subsequently throughout the benefits life-cycle. Useful tools include RACI frameworks,
stakeholder workshops and other techniques to link benefits to organisational priorities.
76
Developing the theme of the benefits life-cycle, there has been some progress towards the
incorporation of an emergent approach to benefits measures and an emphasis on the post-
implementation phase as a focus for benefits realisation. However, this needs to go further.
Therefore,
Recommendation 4 is that guidance needs to emphasise the importance of benefits

measurement and management over the whole life-cycle, taking an emergent approach that
stresses benefits realisation post-implementation.
While there are many areas in which progress is being made, a key challenge remains
accountability for benefit identification and realisation at the Executive and Board levels in
organisations, which is essential if potential long term benefits after handover are to be
achieved. Equally well, operational managers can compromise the achievement of long term
benefits if they do not take ownership for the benefits and therefore do not see benefit
realisation as an integral part of their role, linked to strategic priorities. Therefore,
Recommendation 5 is that the P3M community needs to explore further ways of encouraging
an enterprise-wide culture of value, from the Board through to operations, to enable
potential benefits from investment in change to be realised.
Identification of benefits measures, addressing measurement issues such as quantification and

monetisation, setting targets, tracking benefits, incorporating emergent benefits and
maximizing long term commitment to the measurement and management of benefits all have
many challenges. Some of these are technical, but they mainly concern behaviours and
attitudes. Generally these issues have to be addressed in relation to the opportunities and
constraints in different organisational contexts. Therefore,
Recommendation 6 is that guidance needs to set a framework which enables different

categories of investment and business sectors to adopt approaches to benefits measurement
and management which fit their own context.
The study has found that there is much descriptive evidence on benefits measurement
available, but that there are also many research gaps, some of which might be filled through
inter-disciplinary working with researchers in areas such as change management, investment
appraisal, evaluation and performance measurement/performance management. However,
there is also a need for further research into many areas of benefits measurement and
management. Examples include
- In-depth studies of the whole benefits life-cycle

- Comparative research across different dimensions, e.g. different industrial sectors and
public/private/third sector
77
- Roles in benefits measurement and management, particularly in terms of the variety of
individuals and groups involved in different tasks at different levels
- The use of different benefits dependencies/mapping techniques and the influence of
terminology on their effectiveness.
Therefore,
Recommendation 7 is that opportunities to increase inter-disciplinary collaboration with

allied research areas should be promoted.
and
Recommendation 8 is that efforts to address priority research gaps in the field of benefits
measurement and management be made.
78
7. Appendices (separate documents)
1. Government Sources Literature Review
2. Professional Bodies Literature Review
3. Consultants/Practitioner Literature Review
4. Academic Literature Review – original sources
4a. Academic Literature Review - new sources
79
Measures for benefits realization.
BREESE, Richard, JENNER, Stephen, SERRA, Carlos, THORP, John, BADEMI, Amgad
and CHARLTON, Michael
Available from the Sheffield Hallam University Research Archive (SHURA) at:
http://shura.shu.ac.uk/18957/
Copyright and re-use policy
Please visit http://shura.shu.ac.uk/18957/ and

http://shura.shu.ac.uk/information.html for further details about copyright
and re-use permissions.

Breese-MeasuresAndBenefitsRealization

Uploaded by

Copyright:

Available Formats

Breese-MeasuresAndBenefitsRealization

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Breese-MeasuresAndBenefitsRealization

Uploaded by

Copyright:

Available Formats

Exhibit E

2017 RESEARCH INFORMED STANDARDS

Measures for benefits realization

1. Executive Summary Page 2

2. Introduction, questions asked and summary answers Page 6

4. Findings – Detailed responses to each research question Page 14

6. Conclusion and Recommendations Page 76

NOTE: APPENDICES include all papers considered in the preparation of this

1. What is the ‘state of the art’ in measuring benefits?

- Government body (aimed for comprehensive coverage of documents –67 in total)

Recommendation 4 is that guidance needs to emphasise the importance of benefits

Identification of benefits measures, addressing measurement issues such as quantification and

Recommendation 6 is that guidance needs to set a framework which enables different

- In-depth studies of the whole benefits life-cycle

Recommendation 7 is that opportunities to increase inter-disciplinary collaboration with

1. What is the ‘state of the art’ in measuring benefits?

We will seek to identify literature that maps different stakeholders to stakeholder-

The term 'assess' in the question is taken to include identification, monitoring,

6. What happens with a gap between benefits accountability and project

15-May 15-Jun 16-Jul 16-Aug 16-Sep 17-Oct 17-Nov 18-Dec

Review, clarify and finalise scope

- Government body (aimed for comprehensive coverage of documents –67 in total)

The report has been prepared by

Following this exercise, the number of sources covered was,

- Government body 67 sources (58 in the first study)

- guidance of a normative, ‘how to do it’ nature

Government Bodies Literature Search

67 sources were reviewed, compared to 58 in the first study.

Professional Bodies Literature Search

Consultants/Practitioner Literature Review

The systematic literature review covered

24 sources were reviewed, compared to 21 in our first study.

Academic publications Literature Search

These terms were fed into three databases

 Google Scholar(title only)

Four broad categories of guidance were reviewed

 BRM guidance issued to/by the P3M community.

The sources in each category are:

 Government PPM Benefits Management guidance – OGC/Axelos MSP® (GSR1) guidance

 Project, Program and Portfolio – 10 Sources

In the Business Case & Cost-Benefit Analysis guidance there is

 N. Ireland (GSR34) distinguishes between: Intermediate benefits, “these describe the

Who measures the benefits is not addressed/specified by 48 Sources (many address

 Business Change Manager – MSP (GSR2), OGC 2006 (GSR12)

Other sources highlights the facilitating/coordinating role of PMO staff:

 Portfolio Benefits Manager/Benefits Realization Manager role – in setting the rules

Many sources emphasise the importance of stakeholder engagement in benefits measurement

 Continuous or Regular/Periodic measurement: 3PM Guidance that emphasises

Are measures added over the project? Emergent benefits?

Measurement post-project & when?

 MSP (GSR2) - measurement continues after the program by BAU/operational

 3PM practitioners & managers, Cross-sector, global scope – 15 Sources

Those that recommend combining quantitative and qualitative measures include:

 New Zealand (GSR59): “Qualitative (intangible) categorisation Benefits expressed in

 Bradley (GSR6) argues that giving financial values to non-financial benefits is

Linking benefit measures to Organisational KPI’s/Strategic Objectives is recommended in

Reference to use of standard measures is made by:

 MSP (GSR1) – in the Benefits Management Strategy

Other detailed practical points on benefits measurement include

 Use existing information sources or performance measurement systems - OGC 2005