KPI REPORTING IN THE PUBLIC SECTOR

Keith T Linard1
Director, Ankie Consulting Pty Ltd
GREENDALE VICTORIA 3341 AUSTRALIA
Tel: 0412-376-317
E-mail : linard.keith#@#gmail.com
(Remove # for email)

ABSTRACT:
Performance indicators are key feedback drivers of organisation dynamics. However, the evaluation
literature gives no scientific basis for their selection or validation. An even more significant omission
in the literature is the validation of business rules associated with such indicators. It is presumed
that managers instinctively know the right way to respond to indicator change. Research shows that
managers typically misread delayed feedback effects and their decisions are correspondingly
inappropriate. The balanced scorecard suffers from the above limitations, especially the feedback
interrelationships between indicators.
Recent reports by the Commonwealth Auditor General raise concerns regarding agency performance
management systems which echo concerns going back to the 1980s. It is apparent that there remains
a lack of understanding of performance indicators as well as evidence of loss of corporate knowledge.
In particular, a review of federal, state and local government guidelines, with the exception of NSW
and Queensland, and a review of federal agencies, with some outstanding exceptions such as AusAID
and various NSW agencies, have little conception of the critical importance of cause-and-effect, or
program logic, in the development of their KPIs.
This paper discusses development of a dynamic balanced scorecard based on the system dynamics
paradigm, which accounts for delayed feedback effects. The paper outlines the modelling of causal
interactions within an organisation that impact on output quality.
Keywords: system dynamics; balanced scorecard; BSC; performance indicators ; KPI;
performance management; systems thinking; cognitive mapping; effectiveness audit.

THE HEALTH OF PERFORMANCE MANAGEMENT IN THE PUBLIC SECTOR

Performance Management has improved & is still improving
First, there is no doubt that there have been dramatic improvements in public sector performance
management over the past 2 decades or so since the Hawke-Keating Labor Government initiated a
dramatic objectives-oriented shake up of the federal bureaucracy with its Financial Management
Improvement Program (FMIP), and corresponding action by State Governments.
My comments and criticisms in this paper should be seen as a desire to lift the game even further
rather than a condemnation of the ongoing efforts of so many good people.
Where have we come from?
Management attitudes and management systems have come a dramatic distance since 1980s.
1984 FMIP survey of every senior executive in the federal bureaucracy revealed that 94% saw
as their primary role to ensure that their budget was spent and that their staff numbers were at
the authorised levels. Few saw as a high priority the achievement of program objectives.

1
Keith Linard is former Director of the University of New South Wales Centre for Business Dynamics & Knowledge
Management and a former Chief Finance Officer with the federal department of Finance.
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

A 1985 FMIP survey of the Deputy Heads of 55 federal government departments and statutory
authorities found that fewer than 70% included achievement of objectives amongst their top
seven critical success factors!
the 1985 FMIP survey of management information systems across the federal public sector
found that "no agencies have executive information systems which permit integration of
financial and staffing data . . . with data on outputs."
Today, there would virtually be no public official, at any level, who has not been exposed to the
rhetoric of performance achievement or who, even at a subconscious level, is not oblivious to
organisation objectives. Thus, for example, in 2004, the federal Auditor General found2
All 63 APS agencies included in the audit included performance management as a specified
goal in their planning documents and almost all included it in their certified agreements.
A focus on performance management and improvement has been integral to successive public
service reforms and has been widely supported [by staff].
But still a way to go
A review of federal and state auditor general or program evaluation reports, however, continue to
identify shortcomings in performance management:
Poorly defined or irrelevant KPIs
Poorly defined KPIs, leading to inconsistent interpretation throughout the business
KPIs that are not relevant to the business mission, objectives or stakeholder requirements
Multiple KPIs measuring very similar or highly-correlated things
KPIs that are outside of the control or influence of the business
No causal relationships in KPIs
Frameworks dominated by financial measures
A predominance of lagging measures and / or not enough leading measures
An inappropriate balance between measures of process vs. outcomes
Inconsistent vertical application of KPIs:
Inappropriate level of detail: too much detail in high-level KPIs, or not enough in low-level
KPIs
KPIs that are not cascaded below corporate level, or measures below corporate level not
aligned with Corporate measures
No link between performance measurement and management:
KPIs used solely for reporting with no link to decision-making
No links between KPIs and individual Performance or Development Plans
Incentive based pay absent or not linked to KPIs
Incentive based pay that relies on a single measure, relies solely on financial measures or
relies on measures outside the individuals influence
For example, the 2004 Performance Audit of the federal Auditor-General Performance Management
in the Australian Public Service found:3
Many agencies do not have established systems that relate to, and support, their performance
management systems.

2
The Auditor-General. Audit Report No.6 200405 Performance Audit - Performance Management in the Australian
Public Service. ANAO, Canberra. 2004.
3
The Auditor-General. Audit Report No.6 200405, op. cit..

-2-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Little attempt has been made by agencies to assess the organisational impact of their
performance management system.
It is difficult for agencies to establish a strong link between the conduct of their performance
management systems and improved performance of their organisations.
Up to one-third of responding agencies do not take staff adherence to the APS Values, as well
as the agencys own value systems, into account when assessing performance.
A significant percentage of salary advancement payments, and even bonuses, are unlikely to
reflect superior performance

KPIS AREAS FOR MAJOR IMPROVEMENT & A PRACTICAL WAY FORWARD

In the remainder of this paper I outline my perceptions, based on 30 years intimate involvement in
performance management, of the continuing deficiencies in the performance management paradigm,
and I propose a practical way forward. I emphasise that I am not focusing on management attitudes or
on performance reporting tools, but rather on the fundamental paradigm itself. What should KPIs be
about, how should we select them and how can we validate them. The shortcomings I will focus on
are:
Cause-and-effect chain;
Feedback and delay in process logic;
Basis for selecting performance indicators;
Business rules ignored dimension of KPIs.
The tools I will propose as essential to address these shortcomings are:
Systems thinking;
System dynamics modelling; and
Dynamic Flight Simulator.
I will build this case around the concept of the Balanced Scorecard which, although not widely
implemented in the Australian public sector, has all the elements of current triple bottom line
performance reporting. Accordingly, I ask you to suspend any negative thoughts about BSC and focus
on the fundamental issues I address.

THE FUNDAMENTAL VALUE ADDED OF THE BALANCED SCORECARD

The Balanced Scorecard is an outcomes oriented performance management system that seeks to link
the short and long term activities of an organisation to its vision, mission, and strategy through the
establishment measurable, consensus-driven goals and the development of linked cause-and-effect
performance indicators.
The Balanced Scorecard is more than performance measurement system. It is a management system
that seeks to focus the efforts of staff throughout the organisation towards achieving strategic
objectives. It seeks to give feedback on current performance so that managers will adapt their
priorities towards value improvement and make decisions that give effect to this. It seeks to give
feedback on the key drivers of longer term future performance, so that when the future arrives it has
been long anticipated in managerial priorities and decision making.
The 1997 Management Advisory Board (MAB) report Beyond Bean Counting recommended the
Balanced Scorecard as
a valuable tool for organisations in both the public and the private sectors that wish to drive
a process of strategic change. It can provide a focus through a performance measurement system
that balances the financial and the non-financial indicators of performance. It also harnesses the
strategic vision of the organisation in a holistic view of the key drivers of success, and closely

-3-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

integrates the strategic vision with the operational planning and budgeting activities of the
organisation.4
The MAB went on to say that the balanced scorecard had a number of potential benefits for APS
departments and agencies:
it provides managers with complex information at a glance;
it ensures managers are managing all the important variables not just the easy ones;
it balances time perspectives current performance as well as the drivers of future
performance; and
it prevents information overload by limiting the number of measures used.
A Good Balanced Scorecard
From the writings of Kaplan and Norton, and from the prolific marketing of Balanced Scorecard
solutions by management consultants and software vendors, we know that a good BSC is not simply
a limited list of measures gathered into four categories. Rather, a good Balanced Scorecard
should tell the story of your strategy5, communicating and promoting adherence to the strategy to all
levels of the organisation.
Linked to strategy
Perhaps the single most important contribution of Kaplan and Norton has been to get the message
across that the setting of the performance framework is a crucial corporate responsibility in which the
direct and ongoing involvement of top executives is essential:
in setting the vision, and the strategy for achieving it
in implementing a performance measurement system focussed on strategy
in communicating down the line that executives are using the system.
In essence, the strategies are a series of cause and effect relationships. If you want to achieve X,
what needs to happen to cause that result? The performance measures would then track the trend
of the organisation's progress toward the desired strategies outcome. 6
A good Balanced Scorecard, says Kaplan, tells everyone in your organization, in a single page,
the story of your entire strategy: Every measure is part of a chain of cause-and-effect linkages. All
measures eventually link to organizational outcomes. A balance exists between outcome measures
(financial and customer) and performance drivers (value proposition, internal processes, learning &
growth).7
Based on cause-and-effect relationships
A good Balanced Scorecard will reflect the vertical cause-and-effect relationships for any given
objective and summary measure, in the same way that the causal relationships are reflected
horizontally across the business value chain by the four perspectives. In other words, having
delineated the causal relationship between the Learning/Innovative perspective, the Internal Business
Process Perspective, the Customer Perspective and the Financial Perspective we take each summary
objective/measure and disaggregate them to determine causation.8

4
Commonwealth of Australia, Management Advisory Board, Beyond Bean Counting - Effective Financial
Management in the APS - 1998 & Beyond. PSMPC, Dec 1997. p.54.
5
Renaissance Worldwide Strategy Group. The Balanced Scorecard -- An Overview
http://www.rens.com/viewpoint/papers/scorecard.html
6
Schmid J. The Balanced Scorecard. FCN Reinvention News. 22/5/2000.
7
Kaplan RS, "The Balanced Scorecard", July 13, 1999. http://www.mastersforum.com/kaplan/kaplan.txt
8
Jamesford Consulting, The Balanced Scorecard - A Strategic Management System,
http://www.jamesford.co.uk/the.htm

-4-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Every measure selected for a Balanced Scorecard should be part of a chain of cause-and-effect
relationships that represent the strategy.9
An integral aspect of the cause-and-effect chain, and one to which I will return in discussing adapting
the Balanced Scorecard better to the public sectored environment, is the value chain.
The value chain is a model that describes a series of value-adding activities connecting a
companys supply side (raw materials, inbound logistics, and production processes) with its
demand side (outbound logistics, marketing, and sales). By analysing the stages of a value chain,
managers have been able to redesign their internal and external processes to improve efficiency
and effectiveness. 10
Performance Drivers and Outcomes
A good Balanced Scorecard should have a mix of performance drivers (leading indicators) and
outcome indicators (lagging indicators) of the organisations strategy - a linked set of measures that
define the long term objectives and the mechanisms for achievement. Outcome measures without
performance drivers do not communicate how the outcomes are to be achieved. They also do not
provide an early indication about whether the strategy is being implemented successfully. Conversely,
performance drivers without outcome measures may enable the business to achieve short-term
operational improvements, but fail to reveal whether the operational improvements have been
translated into outcomes achievement.11 The drivers of performance ("lead indicators") tend to be
unique because they reflect what is different about the strategy.
Linked To Financials
A good Balanced Scorecard should (at least in the private sector) link strategy with the desired
financial outcomes. With the proliferation of change programs underway in most organisations today,
it is easy to become preoccupied with a goal such as quality, customer satisfaction or innovation.
While these goals are frequently strategic, they also must translate into measures that are ultimately
linked to financial indicators.12 The fundamental goals of a public sector organisation will be different
(this is discussed later in this paper) but the principle remains valid, that strategy should link with the
fundamental goals.
CAUSE & EFFECT CHAIN - THE CRUCIAL ELEMENT IN THE BSC
The criticality of cause-and-effect to a good Balanced Scorecard is a constant refrain of Kaplan and
Norton. However, few practitioners (managers, consultants or software vendors) appear to have
comprehended the full import of their message. In part this is due to some ambiguity on Kaplan and
Nortons part, as will be discussed shortly.
Is cause-and-effect really important?
Arie de Geus, as head of strategic planning of Royal Dutch Shell through the turbulent decades of the
1970s and 1980s, successfully steered the company through a systemic strategy driven approach.
The research underpinning his approach demonstrated there is something dramatically wrong with
corporate business survival rates13.
A full one-third of the (Fortune '500' list of industrials) listed in 1970 had vanished by 1983
The demographics of companies, their birth and death rates, seem to indicate that their average
life expectancy is no more than 40-50 years. This finding seems to be valid in countries as wide
apart as the USA, Europe and Japan.

9
Ibid.
10
Rayport JF and Sviokla JJ. Exploiting the virtual value chain. Harvard Business Review. Nov-Dec 1995.
11
Renaissance Worldwide Strategy Group. The Balanced Scorecard -- An Overview
http://www.rens.com/viewpoint/papers/scorecard.html
12
Ibid.
13
De Geus, A. Strategy and Learning. Rotterdam School of Management. 1996

-5-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

De Geus argues persuasively that attention to systemic issues (loosely, cause-and-effect relationships),
and in particular to lead indicators through use of business flight simulators is critical to survival:
we will not perceive a signal from the outside world, unless it is relevant for an option for the future
which we have already worked out.
De Geus conclusions parallel the findings, from a different standpoint, of Kaplan and Norton whose
research into company success and failure underpins their design of the Balanced Scorecard, and
especially the cause-and-effect focus.
Cause-and-effect according to Kaplan and Norton
In article after article and example after example, Kaplan and Norton emphasise that the chain of
cause-and-effect should pervade all four perspectives of the Balanced Scorecard and should include all
key performance drivers that ultimately impact on strategy.
Our experience is that the best Balanced Scorecards are much more than collections of critical
indicators or key success factors organized into several different perspectives. The multiple
measures on a properly constructed Balanced Scorecard should consist of a linked series of
objectives and measures that are both consistent and mutually reinforcing. The metaphor should
be a flight simulator, not a dashboard of instrument dials. Like a flight simulator, the scorecard
should incorporate the complex set of cause-and-effect relationships among critical variables,
including leads, lags, and feedback loops that describe the trajectory, the flight plan, of the
strategy. 14 (Emphasis added)
Kaplan and Norton argue that, by having an explicit set of linkages among Balanced Scorecard
measures, managers can test the organisations hypothesised causal chain of performance drivers and
outcomes and can learn how different business rules impact on organisation performance. Whilst
acknowledging that many such hypothesised linkages will be subjective and qualitative, the end point
would be the Balanced Scorecard captured in a system dynamics model that provides a
comprehensive, quantified model of a businesss value creation process. 15 They go on to clarify
what they mean by this, referring to the system dynamics paradigm developed by Jay Forrester at
MIT, quoting various system dynamics references.
In effect, the causal and dynamic relationships in a Balanced Scorecard can be modelled with a
system dynamics approach. 16
Unfortunately, few practitioners appear to have heard this, and fewer still appreciated its import.
Cause-and effect in the general Balanced Scorecard literature
A Google search of balanced scorecard will identify over 7 million hits. Digging deeper will find
about 50,000 of these make some reference to cause-and-effect, which is staggering since the whole
concept of the BSC is related to cause-and-effect. Further analysis will reveal that fewer than 5,000
documents, many of them from academia or software vendors, show any depth of understanding about
cause-and-effect. A laborious review of several hundred US federal, state and municipal government
reports on performance indicators and the BSC revealed almost total lack of substance regarding
cause-and-effect indicators.
Review of federal and state public sector guides on performance management will typically include
some reference to causality, but rarely offer guidance on how to address this. For example, the 2004
West Australian guidelines on Outcomes Based Management simply state: Effectiveness
indicators will generally be derived from some characteristic of the outcome and they should be

14
Kaplan, RS and Norton DP, Linking the Balanced Scorecard to Strategy, California Management Review, Vol. 39,
No. 1, Fall 1996. p.64.
15
Ibid., p. 67.
16
Ibid., p. 79.

-6-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

designed to identify as clearly as possible the causal relationship between the service(s) and the
outcome.17 AusAID is shining exception.
Cause and effect in Australian public sector guidelines
Program logic analysis may be defined as the systematic study of the presumed relationships between
political & regulatory environment, program resource inputs, ongoing program processes / activities,
short term outputs, longer term results, and program objectives.
The 1984, Financial Management Improvement Program (FMIP) Handbook, Evaluating Government
Programs18, proposed a structured logic analysis framework for planning and implementing program
evaluations. The program logic framework was further developed by various State Governments, but
especially NSW, over the intervening years.
A program logic framework remains significant in various federal departmental evaluation manuals
and in the reports and guidelines of the Australian National Audit Office19 and the Federal Department
of Finance and Administration20.
Figure 1 depicts the 2000 Department of Finance and Administration presentation of such a program
logic framework.

Figure 1: Causal Logic Model for Government Program (DOFA 2000)

The logic model provides for the use of output groups that can strengthen the strategic and causal
connections between each level ... The process of developing and analysing the underlying logic of
programs is a powerful mechanism for identifying the key areas and issues within a program,
particularly in relation to outcomes, and hence enables the development of useful performance
information. 21

17
Department of Treasury and Finance, Government of Western Australia: Outcome Based Management - Guidelines
for use in the Western Australian Public Sector. 2004
18
Department of Finance. Evaluating Government Programs - A Handbook. AGPS, Canberra, 1987. pp.13-15.
19
ANAO. Better Practice Guide Performance Information Principles. 01/11/1996
20
Department of Finance & Administration. The Outcomes & Outputs Framework Guidance Document. Nov 2000.
NOTE: The Department changed its name to Department of Finance and Deregulation in Dec 2007.
21
ANAO (1996). Op. cit.

-7-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

The NSW public sector has emphasised a program logic framework for over 20 years, with the
outcomes hierarchy a key tool. The outcomes hierarchy consists of an overarching goal, underneath
which are a set of desired results which contribute causally to the goal. Underneath each desired result
is a set of intermediate outcomes, the achievement of which is presumed to bead to the result. At the
bottom of the chart are the specific services delivered to achieve each intermediate outcome. Arrows
indicate which services contribute to which intermediate results and so forth. This is shown
conceptually in Figure 2.22

Figure 2: Program Logic Outcomes Hierarchy From Services to Goals

Figure 3 shows this tool put into effect by a rural local government body in NSW.23 The analysis
seeks to identify the presumed causal linkages between services provided by the Shire, the
intermediate outputs from those services and their contribution to achieving the corporate goals.

Figure 3: Program Logic Outcomes Hierarchy From Services to Goals

22
NSW Council on the Cost and Quality of Government. Developing Whole of Government Performance Indicators.
2005.
23
Independent Inquiry into the Financial Sustainability of NSW Local Government. Developing whole of sector
performance indicators to improve the accountability, efficiency and effectiveness of local. Dec 2005

-8-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Why Logic Analysis ?

The purpose in undertaking a logic analysis is to describe clearly the intended or assumed processes by
which a service or program is expected to accomplish its objectives. This presumes that the service or
program can be considered, and represented, as a causal system, as illustrated in its simplest form in
Figure 4.

Figure 4: Essence of Program Logic - Causal Chain Leading from Inputs to Outcomes

When presented in this blunt fashion it begs for the question to be asked: How valid are these
presumptions
Most government program activities are, of course, very complex: there are often many interactions
with other programs and with the external social and political environment; the 'causal chain' from
program activities to expected results is often unclear; and the mapping from outputs to outcomes or
objectives is often problematic. Logic analysis inevitably involves simplification of the real world.
Nevertheless, it imparts a salutary rigour to program design, implementation and evaluation by putting
attention of those assumptions which may not hold up in the real world as illustrated in Figure 5.

Figure 5: Logic Analysis Helps Identify Breaks & Dispersion in the Causal Assumptions
This diagram highlights the risk of breaks or dispersion in the logic chain::
what are the key assumptions underlying the program and how will program success be
affected if they are not valid;
what tests or measures might be appropriate to check whether these assumptions are sound;
what are the most sensitive or significant variables and what monitoring and control measures
can we implement to rectify problems as they occur;

-9-
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

what changes to program design, program management or program operations would improve
the likelihood of achieving the desired objectives;
how do various program element activities relate to each other, and how does performance in
one affect the other activities;
what aspects of the program are likely to be affected significantly by other programs or by
factors outside of the program manager's control; and
what are the significant unintended impacts of the program.
Flawed understanding of cause-and-effect
However, both the Kaplan & Norton BSC framework, and the logic framework and outcomes
hierarchy used in the Australian public sector suffer from a flawed understanding of cause-and-effect.
cause-and-effect necessarily involves a time lag between the cause and the resulting effect,
however the BSC and Logic Framework KPIs present the cause and effect indicators at the
same time with no analytical basis to ascertain the implications of the time lag;
the almost universal presentation of cause-and-effect chain is a uni-directional causality which
totally ignores feedback, and especially delayed feedback effects, both within and between the
sectors of the Scorecard.
Time Dimensions of the Scorecard
The Balanced Scorecard presents a static snapshot of the performance drivers, ignoring the fact that
the effects of the measures will occur at different points in time. Thus whilst streamlining the
processing of claims may yield more satisfied clients within a matter of months and is readily
measured, investment in research and development may take years to produce a result. The lack of
apparent impact of the latter on the scorecard may lead to poorly advised decisions.
The significance of ignoring the time dimension, in relation to cause-and-effect, can readily be
illustrated from the project management sphere where, regardless of the nature of a project, managers
always need to know where the project stands in time and cost as measured by the original project
schedule, and the approved project budget. Together these constitute the project baseline.24

Figure 6: Implications of Differences in Reported & Actual Schedule and Cost Variance

When cost variance (CV) and schedule variance (SV) are plotted on the same graph, a bulls eye
diagram is created. Referring to Figure 6, a result plotted at the centre (the intersection of the CV and

24
This example is based on research at the UNSW Centre for Business Dynamics and Knowledge Management by Dr
Alan McLucas.

- 10 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

SV axes) is on schedule and on cost; a result in the top-right quadrant would be ahead of schedule
and under cost; and so on. This simple report shows not only project status, but how that status is
trending. On the surface its appears to address timing issues.
But, like the Balanced Scorecard, this reporting system aggregates snapshot data from different time
periods. It is readily demonstrated that when delays in data collection and reporting occur, and current
information is aggregated with information one or two months old, CV and SV as reported bear little
resemblance to actual progress. This is depicted in Figure 6 (a Defence project) where the reported
CV and SV (suggesting the project is on-time but slightly behind on cost yet trending in the right
direction) are compared to the actual CVact and SVact (which show a totally different and somewhat
alarming picture).
More generally, the dangers associated with ignoring the time dimension relating to cause-and-effect
chains are illustrated by the periodic boom-and-bust in the property market. The market is robust;
returns are going up; it is a good time to invest in new development. Of course, the development
process takes time. In the meantime returns continue to improve, encouraging even more developers
to enter the market. The early developments come on stream and sop up the demand. When later
developments enter the market, demand has already crashed. Investors and developers alike are hurt
and development comes to a stand-still. Eventually the surplus stock is taken up, there are shortages
in supply and prices start to climb. The market is robust; returns are going up and we are back on
the roller-coaster.
We are living with the results of a similar dynamic at the moment as a result of the boom and bust in
the sub-primes marketplace.
Interrelationship amongst performance drivers - uni-directional causality
In most of their writings, the relationship between measures on the Balanced Scorecard is
ambiguously described by Kaplan and Norton. Every example presented and every diagram
describing cause-and-effect present or imply a unidirectional impact, as illustrated in Figure 7. This is
also the typical case with diagrammatic representations of the logic framework or outcomes hierarchy
of government programs, as is evident from Figures 1 to 3..

Figure 7: Example from Kaplan & Norton - Unidirectional Cause & Effect Chain Supporting
Strategy ( Kaplan and Norton, 1996)

The Revenue Growth Strategy The Productivity Strategy

"Reduce the volatility of earnings by broadening the "Improve operating efficiency by shifting customers
sources of revenue from current customers" to more cost-effective channels of distribution"
FINANCIAL
Improve Returns PERSPECTIVE

Broaden Revenue Improve Operating

Mix Efficiency

CUSTOMER
Increase Increase PERSPECTIVE
Customer Customer
Confidence in Satisfaction
our Financial through Superior
Advice Execution

INTERNAL
PERSPECTIVE
Understand Develop Cross-Sell Shift to Minimize Provide
Customer New the Product Appropriate Problems Rapid
Segments Products Line Channel Response

LEARNING
Increase Employee PERSPECTIVE
Satisfaction

Develop Align
Strategic Personal
Skills Access to Goals
Strategic
Information

- 11 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Figure 7 assumes that Employee Satisfaction will not be affected by changes in work processes
(internal perspective), changes which might involve staff reductions, reskilling, higher intensity work,
etc. It assumes that there is no feedback from success in the customer sector to Employee
Satisfaction. Arguably, the (ignored) feedbacks from the Internal and Customer perspectives
would have greater impact on employee satisfaction than the rather rarefied causes suggested in the
Learning perspective. Figure 7 also ignores the obvious feedback link from financial returns which
provides the capacity to invest in learning and growth or reengineering of internal processes.
Misperceptions of delayed feedback dynamics- the critical issue
An implicit and fundamental assumption behind the Australian public sector approach to selecting
performance indicators is that the feedback they give to the decision maker (either directly or via
resultant pressure or direction from others) will cause the decision maker to make appropriate
adjustments to the inputs or processes.
However, there is abundant research in the field of system dynamics25, as well as in the fields of
experimental economics and psychology which suggest that managers have great difficulty managing
dynamically complex tasks. Sterman argues persuasively from his work at MITs Sloan School of
Management, that there is systematic misperception of feedback especially when there are delays in
the system. Mosekilde, Larsen and Sterman26 present the results of 48 simulations of the Beer
Game27 (a simulation of a simple factory-warehouse-retail system) run with 192 graduate students
from MIT and senior executives of major US firms. They show that decision making on the basis of
straight forward performance indicators, but in the face of delays, consistently resulted in costs
averaging more than 10 times the optimum!
Simulations run by the author over 14 years at UNSW involving undergraduate and graduate students
as well as many senior state and federal public sector managers showed a similar pattern. In both the
MIT simulations and those at UNSW, highly educated managers and students failed to comprehend
the significance of feedback in the face of delay induced dynamics.
In other experiments at MIT, where graduate students had full information, training, incentives and
opportunities for gaining experience, Diehl and Sterman still found poor managerial performance in
the face of variations in feedback strength and delay.28 Often they found the subjects were
outperformed by a simple no-control rule. Diehl and Sterman argue that the mental constructs and
heuristics that managers bring to bear on complex tasks are fundamentally dynamically deficient:
Subjects were unable to account well for delays and feedback effects because (1) peoples mental
representations of complex tasks are highly simplified, tending to exclude side effects, feedback
processes, delays, and other elements of dynamic complexity; and (2) even when these elements are
known, peoples ability to infer correctly the behaviour of even simple feedback systems is poor.
The first deficiency can certainly be addressed through training. The second, however, ... is a
fundamental bound on human rationality - our cognitive capabilities do not include the ability to solve
systems of high-order non-linear differential equations intuitively.

25
Sterman, J. Modelling Managerial Behaviour: Misperceptions of Feedback in a Dynamic Decision Making
Experiment. Management Science, 1989, 35(3), 321-339.
Smith, V, G Suchanek and A Williams. Bubbles, Crashes and Endogenous Expectations in Experimental Spot Asset
Markets, Econometrica, 1988, 56(5), 1119-1152.
Funke, J, Solving Complex Problems: Exploration and Control of Complex Systems, in R Sternberg and P Frensch
(eds.), Complex Problem Solving: Principles and Mechanisms. Erlbaum Assoc., New Jersey, 1991.
26
Mosekilde, E, E Larsen and J Sterman. Coping With Complexity: Deterministic Chaos in Human Decision making
Behaviour, in J Casti and A Karlqvist (eds.), Beyond Belief: Randomness, Prediction and Exploration in Science.
CRC Press, Boston, 1990.
27
The Beer Game is described in detail in Senge, P, The Fifth Discipline - The Art and Practice of the Learning
Organization. Doubleday, New York, 1990
28
Diehl, E and J Sterman. Effects of Feedback Complexity on Dynamic Decision Making. MIT Sloan School of
Management, Research Report D-4401-1. March 1994.

- 12 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Simulating the Beer Game

The so-called Beer Game (more precisely an inventory management game) is a classic management
game developed at MIT to simulate decision making in the face of dynamic processes, testing the
systems principle that "structure influences behaviour. The scenario for the MIT game is depicted in
Figure 8. Each team comprises 4 sets of decision makers - an ordering executive and a supply
executive for each of the retail, wholesale, distribution and factory sectors. The aim of each team is to
minimise total inventory costs and stockout costs, in the face of delays, as indicated in the diagram,
and of demand uncertainty. As noted earlier, even with such a simple system senior executives and
MBA students are unable to account for the dynamic interactions.

Figure 8: System simulated in the 'Beer Game'

In this game, the customers initially purchase 4 items per day. At, say, week 10, the orders jump to 6
items per day and thereafter remain constant. If the management team applies a simplistic Only
order what you have sold heuristic, the stock pattern will look similar to Figure 9. The decline in
inventory occurs because there is a two week delay before the higher level of sales (which deplete
inventory) are translated into a higher level of orders (which replenish inventory).

Figure 9: Beer Game Stock Pattern Over Time without KPIs

The putative performance management panels are now challenged to implement performance
indicators and related decision rules which will force inventory to tend towards target. Group
brainstorming with senior managers and graduate student groups (reminiscent of public service
performance indicator workshops) inevitably results in decision rules which, as we see in Figure 10,
lead to disaster. The players almost invariably set a single performance indicator (Target_Inventory
less Actual_Inventory) and set the decision rule: order what was sold plus the performance gap.
In the face of a simple perturbation (a once-off increase in sales) and a short delay, this intuitive
indicator sets up chaotic fluctuations. The mix of performance indicators, and the related decision

- 13 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

algorithms required to force the system to stability at the target inventory is by no means intuitive,
notwithstanding the problem simplicity.

Figure 10: Beer Game Stock Pattern Over Time with KPIs
CHOOSING INDICATORS AND VALIDATING BUSINESS RULES
A fundamental purpose of performance indicators is to give decision makers feedback on program
operations in order to guide future decisions. In order to be confident regarding decisions made
consequent on such feedback, however, two key conditions must be satisfied:
there should be a rigorously basis, beyond mere consensus or availability, for selecting
indicators); and
there should be a way of developing and validating the business rules appropriate to off-
trend movements in these decision criteria.
Unfortunately the Balanced Scorecard literature is singularly silent on these points.
Basis for choosing indicators
Ive looked at hundreds of Australian public sector documents seeking guidance on the selection and
specification of KPIs. Unfortunately all one gets are truisms or platitudes:
1. indicators exist to assist decision making: an important purpose of performance indicators is
to provide feedback to the managers to guide decisions on continuous program improvement;
2. there are a variety of indicator types (input, process, output, outcome), but performance
indicators should focus on a limited number of key outcome areas designated by program
managers as critical to the continuing successful functioning of the agency;
3. a first step in selecting indicators is ensuring that the objectives are appropriately stated;
4. there should be widespread managerial and staff involvement in the selection of the
performance indicators so that ownership will occur;
5. there are a variety of uses for indicators, including strategic and program planning, leadership
and motivation, management control, accountability and evaluation;
6. there are a variety of audiences for performance indicators, including agency managers at
different levels, executive Government, Parliament, lobby groups, press and public.
Beyond these platitudes, the public sector literature has little to say on rigorously identifying and
validating performance indicators. The assumption is that indicators are susceptible to the democratic
process. The wider literature on performance management is similarly unhelpful in moving beyond
vague platitudes.
Business rules for acting on indicator feedback
Even if we have the right KPIs, the management literature is essentially silent on the issue of
validating business rules for responding to off-trend indicators and of training managers in this regard.

- 14 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

The assumption is that when the red light starts flashing, managers will know intuitively which
levers to push and in which direction. This is manifestly fallacious.
Whilst the Chernobyl and 3 Mile Island nuclear disasters are possibly the most striking examples of
the fallacy of this assumption, there have been a host of major corporate bankruptcies over the past
decade which attest to the significance of this point. Let me illustrate the significance of the issue.
The original VW Beetle, with its engine in the rear, was very susceptible to going into a rear-wheel
skid when cornering. Picture the scene: Our digital dashboard KPI system flashes Rear-Wheel
Skid. The manager (driver) has 3 decision levers: the brake, the accelerator and the steering wheel.
The instinctive managerial response is to steer out of the skid and / or hit the brake. The technical
reality is that either of these business rules will accentuate the skid; whilst application of both could
cause the car to roll. The correct business rule is to turn into the skid whilst gently accelerating, and
only after regaining control, turn out of the skid.

Knowing a problem exists and knowing the decision levers available does not guarantee that the right
lever(s) will be pushed in the right direction.
The assumption that a good KPIs will necessarily result in managers making good decisions based
on indicator feedback is debateable. There is considerable research to suggest that the assumption is
invalid, especially in the face of delayed feedback.

INDICATOR HIERARCHY AND CAUSAL INTERRELATIONSHIPS

The following sections suggest tools and methods to address the shortcomings, discussed above, of
current approaches to implementing the Balanced Scorecard:
identifying potential KPIs and strategies
identifying interrelationships between KPIs and strategies
using flight simulators to train managers on how to respond to KPIs
Cognitive mapping technique to identify perceived +ve & -ve relationships
Cognitive mapping derives from the Psychology of Personal Constructs developed by cognitive
psychologist, George Kelly29. It is a powerful and effective tool to capture and analyse stakeholder
concepts about strategic issues. Experience in many environments, including the Australian Defence
Organisation, demonstrate that it can result in high levels of shared understanding among stakeholders,
analysts and decision makers.
A cognitive map, concept map or causal map consists of nodes, known as concepts linked by
arrows. The arrows may carry a sign, though the absence of a sign is usually taken to indicate a
positive link and a negative sign indicates a negative link. Causality feedback and dominant

29
Kelly, G. A. (1955). The Psychology of Personal Constructs: A Theory of Personality. Norton, New York

- 15 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

mechanisms may be derived from these maps. McLucas has developed a robust methodology, within
an Australia context, for the application of this technique.30
Figure 11 illustrates the end point of a cognitive mapping process focusing on elements of the Defence
Science and technology Organisation (DSTO) draft Balanced Scorecard. The draft BSC exhibited the
traditional uni-directional cause-and-effect chain. The cognitive mapping process, combined with a
cluster analysis, not only revealed profound inter-relationships, but highlighted several negative
relationships (dotted lines) which had not previously been appreciated. (A negative relationship
suggests that a particular strategy designed to promote achievement of one strategy, has a side effect
of countering another strategy a possibility ignored in the BSC literature.)

Figure 11: DSTO KPI Influence Diagram Developed from Cognitive Mapping Process

Clustering techniques for consolidating indicators

The Balanced Scorecard literature emphasises the importance of focusing on a small number of key
performance indicators, but again gives not suggestions on how this selection might be done.
Statistical cluster analysis provides a simple and robust technique. In essence, cluster analysis groups
like factors with like, and separates unlike from unlike.
Cluster analysis can be used at various stages in developing a Balanced Scorecard. For example, in
relating KPIs to strategy, the management team (individually or collectively) assigns to each KPI a
score (e.g., on a scale of -10 to +10) based on judgement of the degree to which a given KPI supports
(1 to 10), is neutral (0) or detracts from (-1 to -10) a given strategy. Figure 12 illustrates part of the
output from the DSTO study.

30
McLucas, A. Qualitative and quantitative techniques for addressing systemic complexity in the context of
organisational strategic decision making. PhD Thesis. UNSW. Aug 2001

- 16 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Figure 12: Worksheet for DSTO Clustering Exercise

The results of such an exercise, e.g. on an Excel Spreadsheet, can then be fed directly to a cluster
analysis program (standard with most statistical software packages). A variety of output formats are
possible, including the dendogram tree in Figure 13.

Figure 13: Dendogram Produced by Cluster Analysis

The dendogram identifies clusters of (in this case) KPIs, providing a rational basis for the
development of the performance indicator hierarchy in the Balanced Scorecard and for the analysis of
inter-relationships.

SYSTEM DYNAMICS AND PERFORMANCE INDICATORS

As noted earlier, Kaplan and Norton argue that the Balanced Scorecard should be based on the
metaphor of the flight simulator, not a dashboard of instrument dials. Like a flight simulator, the

- 17 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

scorecard should incorporate the complex set of cause-and-effect relationships among critical
variables, including leads, lags, and feedback loops that describe the trajectory, the flight plan, of the
strategy. 31
System dynamics software enables just this, the creation of management flight simulators which
integrate with the corporate Balanced Scorecard system. Managers can learn how performance
criteria affect not only their decisions but the performance of colleagues and of the organisation. They
can also test the longer term implications of different decision rules.
The use of dynamic models, particularly in a gaming environment, adds critical dimensions to the
Balanced Scorecard. The following section discusses a prototype dynamic balanced scorecard flight
simulator designed to enable managers to test drive performance indicators and related business
rules.

BUILDING A PUBLIC SECTOR DYNAMIC BALANCED SCORECARD

The essential purpose of the various state & federal government outcomes and outputs framework is to
answer three questions:
(i) What does government want to change? (outcomes)
(ii) How does it want to make that change? (outputs)
(iii) How does it know if it is succeeding in making that change? (indicators)
The system works as a hierarchy. Government, through its ministers and with the assistance of
relevant agencies, specifies the outcomes it is seeking to achieve in a given policy arena. These
outcomes are specified in terms of the impact government is aiming to have on some aspect of society,
the economy or the national interest. Parliament appropriates funds to allow the government to
achieve these outcomes.
These funds allow government to buy outputs or services from agencies. Agencies specify the
outputs (e.g. policy advice) they undertake to provide to contribute to the achievement of these
outcomes, including through third parties.
Specification of what will be delivered can help improve the understanding and knowledge of those
outside the agency who have an interest in its performance, including Ministers, Parliament and
external accountability bodies such as the Auditor General.
The framework encourages agencies to focus on the specific products they deliver (outputs) and how
to deliver them in the most efficient and effective ways; and to identify business lines they should
drop or outsource. The implementation of accrual accounting using generally accepted accounting
principles provides the basis for benchmarking comparisons so that decisions about preferred
providers are better informed.
Where services are provided direct to industry and the community, the development of client service
charters is an integral part of the reforms and an important indicator of performance. The
Government itself makes decisions on programs and (usually) on guidelines for service provision.
Within these guidelines, however, governments expect service delivery to be of a high standard and
public servants are accountable to their Minister and to government for their service to customers in
this context.
Charters must specify standards for service delivery and set out complaint handling and feedback
mechanisms, and must be developed through consultation with customers and staff. Agencies are
required to report annually, through the Department of Finance and Administration, to the Parliament
on their performance against the charter.
The Customers of the Federal Public Sector
The process outlined above provide a more complex view of the customer perspective for the
Australian national public sector then that which exists in the private sector. In the public sector

31
Kaplan,RS and Norton DP, Linking the Balanced Scorecard to Strategy, California Management Review, Vol. 39,
No. 1, Fall 1996. p.64.

- 18 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

context, the prime customer (i.e. the agent who pays for a reciprocal service) is the Minister on behalf
of the Government.
Departments also have a legal requirements which set up customer relationships, especially to the
Auditor General and to the Parliament in respect of governance, especially in relation to fraud
prevention. Other management oriented legislation, such as Equal Employment Opportunity and
Occupational Health & Safety legislation, apart from their moral imperative, have the potential to
create significant embarrassment for agency management and the Minister if they are breached. These
clearly create a distinct customer.
In those cases where Federal Departments do provide services direct to the public, they are required to
prepare and implement a service charter, providing a clear customer relationship.
Thus it is suggested that are four Customers who may need to be addressed in a BSC:
for most departmental activities, the Minister, and through him/her, the Government;
in respect of governance, the Auditor General and Parliament, as well as the Minister;
in respect of organisational health (OH&S, EEO etc) Parliament, the Minister, staff
for service delivery activities, the corporate or individual service recipients.
These and other considerations have led us to propose that a 7 Sector Balanced Scorecard, depicted in
Figure 14, may be more appropriate for public sector agencies that the classic 4 sector model.

Resource Management
Client Orientation
GOALS Governance & Probity
Organisational Health
Structure & Processes
Organisational Learning
Corporate Communications
Figure14: Seven Sector Balanced Scorecard for Government

Dynamic Balanced Scorecard - Integrating System Dynamics & BSC

The following material summarises research done in a number of government agencies designed to
build this seven sector dynamic Balanced Scorecard
The BSC template that has been developed focuses specifically on the relationship between
managerial responses to any workload-resourcing gap (the Capacity Gap), and how this impacts on
the first two Customers above. The model outputs relate specifically to output quality index;
probability of fraud / probity incidents; and probability of major OH&S / EEO incidents.
The following causal loop diagrams illustrate the mechanisms by which these are impacted by
departmental operations. The interrelationships between the respective work area outputs and their
impact on achievement of outcome indicators is subject of a separate study.
Resources Performance Management in the Public Sector
All federal Government CEOs consider financial results to be one of the top three indicators of their
organisations success. However, financial management in these agencies differs dramatically from
the private sector context in that the revenue side of the budget is a given, and the focus is simply on
effective and efficient management of the expenditure.
The quantum of funds made available to a Minister to implement Government programs is the end
result of a complex interplay of macro-economic deliberations, ministerial bargaining and political
judgement. Issues regarding what can be delivered (in terms of quality and quantity) for the proposed
funds are significant inputs. But such relationships are largely approximations. Lack of skilled

- 19 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

management (for example as a result of high turnover or overwork) can result in errors in estimating
required workload.
When the Government has determined its policy, the bureaucracy has only minor leeway in changing
the quantum or time schedule of the service. In addition, a variety of unplanned business pressures
inevitably impact on planned business. Unforeseen events such as fraud within the Department, a by-
election in a sensitive electorate, or a major controversy relating to the Ministers policy responsibility
inevitably generate workload which is expected to be absorbed. The resource management task is to
deliver the planned outputs within budget.
Figure 15, overleaf, illustrates interrelationships in the resource sector. Customer Demand
represents the Governments (through the Minister) expectations which also determines the resourcing
level. It is almost axiomatic that available resources will be less than that required for quality
implementation of all the planned workload, let alone the inevitable unplanned demands. Whether the
management response to any capacity gap is innovative or dysfunctional is a function of the
organisational competencies, which in turn is the result of leadership and investment in capability, the
latter balancing short term impacts on recurrent resources.

Department s
Resources
Customer
s Demand
o s s

Capability Recurrent
Investment Resources Base Load 'New'
Business Business
s
o

Total s Annual s
Effective s Workload
Capacity o Planned s
o s

CAPACITY GAP Business

s
Planning
Errors
s

o
Dysfunctional s Innovative
Responses to Responses to Management
Gap Gap Effectiveness

o s
s
Learning Organisation s
Competencies

Figure 15: Key Resource Sector Interdependencies in Public Sector Agency

Figure 16 depicts the staffing module of the Resources Sector in the Dynamic BSC model. This
model tracks skill levels of senior executives, executives, technical and administrative support staff,
based on staff turnover rates and time at each level to reach full efficiency.

- 20 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Resources Return To
Model
Personnel Sector

NoviceAttRate TrainAttRate_2 CompAttRate_1 SkilledAttRate_1

NovAttRate TraiAttRate CompAttRate SkilledAttRate

!
hiring_SupStaff progress1 progress2 progress3 progress4 departing_
! ! !
Novice_ Trained_ Competent__ Skilled_ Experts

Progress_delay Progress_delay Progress_delay Progress_delay Attriton_Rate

Support_Staff_Need InitCompetent
InitNovice InitTrained InitSkilled InitExpert

Executive_Staff_Need

hiring_ExecStaff

Figure 16: Staff Resources Segment of Dynamic Balanced Scorecard Model

Key resource management performance criteria relate to corporate governance (probity, fraud control,
etc), managing expenditure patterns (and especially employment related expenditure patterns) to
budget and achieving efficiencies in an increasingly efficient environment.
The Characteristics of the Internal Processes in the Federal Public Sector
The core processes in a public sector agency are essentially the same as for the private sector. These
are to:
Establish direction;
Acquire resources;
Provide capability; and
Execute the mission.
Whilst the Government establishes the policy and program outcomes that are to be achieved in
exchange for the financial resources, management translates the vision and allocates the capabilities to
achieve the delivery of agreed outputs. The resource management framework provides a backdrop for
an integrated planning process that links corporate plans, business plans and individual plans. This
planning process focuses on achieving results through the delivery of outputs as the agency, the
business unit and the individuals performance is linked to the outputs which in turn is linked to the
outcomes that the Governments desires for the community.
As noted in relation to resource management, once the budget framework is set, Federal Departmental
managers have limited scope for obtaining extra resources. Any increase in workload, or workload
underestimate, typically will be addressed by working harder:
more intense and longer hours of work (unpaid overtime)
reduction in time devoted to training and development
reduction in strategic management activities (through redirection of 'management time' to 'task
time')
deferring some work (which simply postpones the day of reckoning) and
reduction in target quality of inputs (e.g. through cutting background research effort) or
outputs

- 21 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

As illustrated in Figure, if such responses are prolonged, they tend to bring about dysfunctional
feedback effects which eventually increase the capacity gap, through increased re-work, falling moral,
increased staff turnover.

s Total
o Effective
Capacity o

s
CAPACITY GAP
Staff
Turnover o
o s s

Work Average Target Output

s Workload
Deferred Quality
Staff
Morale
s o
Productivity
s s
o

Work Re-work / o
s Intensity Errors

s s Time on
Strategic
Management
Management
s
Effectiveness

Figure 17: Key Interrelationships Impacting on Internal Processes

Where there is excellence in leadership and a culture characterised by the learning organisation one
could expect innovative responses to any significant or prolonged capacity gap. In essence such
innovation changes the rules of the game and achieves the end result much more efficiently. There
are many local examples of such behaviour across the bureaucracy, but at this juncture the authors
would be sparing in their application of learning organisation to Departments as a whole.
Only dysfunctional responses, as illustrated in Figure18, have been incorporated in to the model at this
stage because of lack of data to permit modelling of the learning organisation response.
Figure 18 is a simplified extract of the Powersim dynamic BSC model relating to the Internal Process
Sector, corresponding to the causal loop diagram above. The module currently draws its base data
from spreadsheet, including:
annual work plans
new policy and extra ministerial projects
average monthly patterns of work
The model also includes allows for outsourcing activities ( not included in the causal loop diagram).

- 22 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Figure 18: Internal Process Sector of the Dynamic BSC Model

In the models flight simulator panel (Figure 19), how the capacity gap is addressed is a matter, each
three months, for user decision, with the user being given the choice of, e.g.,:
specifying a % of the excess workload to defer to the next year
switching a % of strategic management time onto the excess workload
switching a % of staff development time on to the excess workload
absorbing the excess workload through reduction in output quality

Figure 19: Dynamic BSC Simulator Control Panel

- 23 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Innovation and Learning in the Public Sector

Recruitment standards to the federal Public Service are high. Almost 50% of all recruits have a
Bachelor Degree or higher. Staff receive support to upgrade qualifications. Middle & senior
management training has a high priority. At the same time, work pressure on all staff, and unpaid
overtime, has increased significantly which acts as an impediment to self-development. Also, in the
face of work pressures, informal and formal staff development tend to be among the first areas to be
constrained. Figure 20 shows key causal relationships impacting on the learning and growth sector.

Capability Recurrent s
Total
Investment Resources
Effective
Capacity o
o s

CAPACITY GAP
Staff
o
Turnover
s

Productivity Average
Staff
Workload
Morale

s s
Re-work /
o o
s Errors

Skill Time for o

Level s Staff
Development
Innovative
Responses to
s s
Gap

Learning Organisation s
Competencies

Figure 20: Key Interrelationships Impacting on Learning & Growth

A critical factor in learning and growth is staff turnover. What little research is available suggests that
turnover rates in excess of 15% are dysfunctional. A number of key government agencies at both
federal and state level have staff turnover at the executive and senior executive levels of almost 30%
per annum, and over 20% per annum across all staff. The model reflects this in loss of productivity.
Corporate Governance & Corporate Health
Figure 21 depicts the Corporate Governance sector of the dynamic scorecard. We have been
undecided whether corporate governance and corporate health should be regarded as sectors in their
own right. On the one hand, Corporate Governance might be viewed as a key indicator in the
Customer Sector, whilst Corporate Health might be an indicator for the Internal Process Sector. On
the other hand, failures in these areas tend to have major political implications for the Minister and the

- 24 -
 Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Government as well as significant feedback interrelationships with the other sectors. This suggests
that they might appropriately be considered as sectors in their own right.
The Australian National Audit Office (ANAO) defines corporate governance to encompass authority,
accountability, stewardship, leadership, direction and control.32 The dynamics BSC model, at this
stage, uses fraud events as a general surrogate for the effectiveness of governance across the broad
range of departmental activities.

Corporate Governance Sector

Fraudulent and Dubious Practices Public Awareness of Fraud

Public_Awareness_of_Fraud

Random_Num
Fraud_Events_MAvg Effect_of_Fraud_Total_on_Aware

Increase_in_Fraud_Prob Fraud_occuring Fraud_Accumulating

! ! ! !
PROBABILITY_OF_FRAUD FRAUD_EVENT FRAUD_TOTAL

ffect_of_Morale_on_Fraud
MORALE

Effect_of_StaffDevelopment_on_Fraud in_progress

StaffDevelopment_Index Return To
Model
Management_Effectivness Sector

Effect_of_Fraud_Total_on_Awareness

Internal Factors Impacting on Fraud

Figure 21: Corporate Governance Sector of Dynamic Balanced Scorecard

Model Output
At the time of writing this paper, only illustrative reporting screens have been developed. Figure 22 is
an example. It is apparent from discussions with departmental staff that ultimate use of the simulator
will depend as much as anything on the look and feel of the output screens. The current version of
Powersim has limited flexibility in screen design and it is likely that custom screens will be developed
in Visual Basic or C++.

32
ANAO, Applying the Principles and Practice of Corporate Governance in Budget Funded Agencies, Australian
Government Publishing Service, Canberra 1997.

- 25 -
Institute of Chartered Accountants ACT Business Forum May 2008
KPI Reporting in the Public Sector

Figure 22: Illustrative Output Screen for the Dynamic Balanced Scorecard

FUTURE DEVELOPMENTS
This paper has highlighted some serious theoretical and implementation shortcomings of KPIs in
general and the Balanced Scorecard in particular, especially in relation to their implementation in the
public sector. Solutions have been outlined to these. In particular, it is suggested that system
dynamics modelling can be used, to provide management flight simulators which incorporate the
complex set of cause-and-effect relationships among critical variables, including leads, lags, and
feedback loops that describe the trajectory, the flight plan, of the strategy.
____________________________________________

- 26 -