See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318398449

Applying business analytics for performance measurement and management.

The case study of a software company

Article · May 2017

DOI: 10.3280/MACO2017-002006

CITATIONS READS

0 909

1 author:

Franco Visani
University of Bologna
21 PUBLICATIONS   110 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Business Performance Analytics (BPA) View project

Supply Chain Management Accounting View project

All content following this page was uploaded by Franco Visani on 12 October 2017.

The user has requested enhancement of the downloaded file.

 The following paper is a pre print version. You can find the

version published in Management Control at the following URL:

https://www.francoangeli.it/riviste/Scheda_Rivista.aspx?IDArticolo=59486&Tipo=Artic
olo%20PDF&lingua=it&idRivista=166

Applying business analytics for performance measurement and management.

The case study of a software company.

Franco Visani

Department of Management - University of Bologna

Abstract
The concept of Business Performance Analytics (BPA) represents the adoption of analytical
methods to support strategy control and (re)formulation. The purpose of the paper is to
explore the potential support that BPA can provide for performance management systems.
The study, through an interventionist approach, empirically tests a 5-step BPA framework
(Raffoni et al. – forthcoming -) and provides insights for further developments.
The longitudinal study, conducted at a company operating in the data-security industry,
supported the analysis of BPA potential and offered a number of insights to refine the
framework and shed light on critical organisational issues. This paper offers both theoretical
and practical suggestions. It is one of the few attempts to investigate the potential of Business
Analytics in enhancing strategic control. It provides insights in a practical framework that
could be used to guide further research in the area and provides an account of potential
challenges faced by businesses when trying to move to strategic control with a data-driven
approach.

KEY WORDS: Business Analytics, Performance Measurement Systems, Business

Performance Analytics, Big Data.

1. Introduction

In today's business world a huge amount of internal and external data is available to
companies. The word "Big Data" represents this high-volume, high-velocity, high-variety and
high-veracity information. Globally produced data are expected to double every 18 months,
with data volume processed by organizations expanding by 35-50% per year (Manyika et al.,
2011). Technology is changing business rules, and how to transform data into knowledge has
become a key issue (Davenport et al., 2010). A number of studies have advocated the
potential contribution of analytical approaches for strategy formulation, decision-making,
support, performance management and measurement (CIMA 2014, Bhimani and Willcocks,
2014; Warren et al., 2015, Raffoni et al. –forthcoming-). The idea that data analysis plays a
crucial role in performance management precedes the analytical phenomenon. The literature
1
points out that strategic data analysis is as important as understanding organization strategic
objectives, allocation of resources and the definition of formal and informal aspects of their
Performance Management Systems (PMSs) (Schreyogg and Steinmann, 1987; Simons, 1995;
Kaplan and Norton, 1996; Ittner and Larcker, 2005). Its contribution is seen as a crucial
element for both the monitoring of strategic success and the promotion of organizational
learning (Schreyogg and Steinmann, 1987; Simons, 1995). Nevertheless, research in this area
is still under-developed and little is known of how the mechanisms of data analysis may
influence the controlling of strategy (Nudurupati et al., 2011). As a result, different issues
emerge with respect to the role of PMSs in controlling strategy. When the target is to control
the implementation of deliberate strategies (Mintzberg, 1978), the PMSs in practice have
highlighted a number of limitations: difficulties in capturing the key business success factors,
unclear impact on business performance, a focus on the past and on the short-run. The use of
intuitive rather than rigorously, evidence-based approaches has been linked to such issues
(Brignall, 2002; Norreklit, 2000; Ittner and Larcker, 2005). Besides, data plays a significant
part when PMSs are used to promote strategy (re)formulation and learning (Galeotti et al.
2016). The higher level of uncertainty requires increasing levels of information processing;
thus the way data is shared, discussed, and debated acquires a prominent role. While extant
research has mainly explored the "degree of interactivity" of PMSs in relation to innovation
(Bisbe and Otley, 2004; Bisbe and Malagueno, 2009), uncertainty (Janke et al., 2014) and
specific capabilities (Henri, 2006), the understanding of the underlying data flow is still
limited (Ittner and Larcker, 2005). Some recent contributions have begun exploring the
potential of Business Analytics (BA) and Big Data for performance management (Warren et
al., 2015; Klatt et al., 2011). Even if some papers hypothesize the potential support of BA for
performance management purposes (Warren et al., 2015), very few studies deeply investigate
how it can be framed and what happens in the real world.
Two notable exceptions are Schläfke et al. (2012) - that presents a first practice-oriented
framework for operationalizing the role of BPA, and Silvi et al. (2012) that by surveying a
group of large and middle-sized Italian companies have observed a relationship between the
use of BPA and more advanced and effective PMSs.
Our work may be placed in this emerging stream of research and aims at theorizing the
role of BA and Big Data for performance management purposes. In light of Simons's notion
of diagnostic and interactive use of PMSs (Simons, 1995), we tested a framework developed
by Raffoni et al. (forthcoming), which discussed a series of issues to be considered when
integrating analytical methods with the PMSs. The term "Business Performance Analytics"
(BPA) is here adopted to represent the controlling of business dynamics and performance
through the systematic use of internal and external data and analytical methods (Silvi et al.,
2011).
The framework is empirically tested in a company operating in the data-security industry
by adopting an interventionist approach (Kasanen et al., 1993). This allowed the researcher
more flexibility in moving back and forth from theory to data, in a relatively novel area of
study, and contributed to a more in-depth understanding of the issues concerning its
application.

2
 To summarise, this study makes three broad theoretical contributions. Firstly, by focusing
on the role of evidence-based decision-making, it extends previous research that has explored
the role of data analysis in the context of performance management (Ittner and Larcker, 2005).
By considering both the diagnostic and interactive function of BPA, it also addresses the call
for a better development of theories on the use of PMSs (Ferreira and Otley, 2009).
Secondly, it tests a practical framework on BPA. A series of issues are identified and then
explored in relation to the use of BPA for performance management purposes and some
critical points, for a further development of the framework, emerged.
Thirdly, the paper contributes to the emerging literature on interventionist research in
management accounting, thus addressing the call for further exploration of the potential of
this approach for practice-relevant management accounting theory (Lukka and Suomala,
2014). Our work provides a detailed account of the interventionist process and highlights the
benefits and challenges of the "insider" role played by the researcher.
The structure of the paper is as follows. The next section presents the theoretical
framework and the BPA model tested. The third section outlines the methodology applied
and the data collection process. Section 4 presents the case study and the empirical findings.
The last section discusses the results in the light of the theory and raises a number of
managerial opportunities.

2. Background

2.1. Business Analytics, Big Data and performance management

The term Business Analytics has become popular after the publication of Tom Davenport's
book "Competing on Analytics: The New Science of Winning" (2007), even if the adoption
of mathematical and statistical analysis on business data to support the decision- making
process dates back to the 1960s (Holsapple et al., 2014; Watson, 2011). At first used in
finance to evaluate credit risk and stock option pricing, the first BA evolved in more complex
system-targeted to support a whole set of business decisions. Starting by the end of the 1970s,
a series of Decision Support Systems (DSS) was developed to support managers in making
decisions in different areas: from finance to operations, from supply chain management to
customer relationship management (Holsapple, 1996; Power and Sharda, 2009). Finally, the
term Business Intelligence (BI) became common in the 1990s to indicate software packages
able to manage structured databases to perform a wide range of actions like reporting, drill
down, slice and dice, or simulations (Moss and Atre, 2003). These systems can support
operations modelling in real time (Berry et al., 2009) and enhance decision-making, either
through the integration of different databases (Chapman and Kihn, 2009) or by offering new
information (Hyvönen, 2007).
BA encompasses different kinds of software, tools and approaches aiming (1) to analyse
and deeply describe a business phenomenon -so called descriptive analytics-, (2) to support
forecasting and predictions -predictive analytics- and (3) provide effective action patterns -
prescriptive analytics- (Evans, 2013). The last step in the evolution of BA is related to the
diffusion of Big Data Analytics (BDA). BDA, which emerged after 2010, allowed the
3
analysis of previously unmanageably enormous unstructured datasets, mainly coming from
outside the company and often from social networks, like Facebook, Twitter, LinkedIn,
personal blogs, etc. The well-known 4 Vs distinguish BDA from previous BA: volume (the
size of the databases raised from terabytes to petabytes, exabytes and even zottabytes, a
volume equal to one billion terabytes), variety (the data are captured from many different
sources, both internal and external), velocity (the data change at an unbelievable rate,
requesting an analysis of streaming data), and veracity (data are uncertain and unstructured).
BA can deliver value along the business value chain involving different organizational
units and departments:

- marketing: cross-selling, location based marketing, in-store behaviour analysis,

sentiment analysis (Maule et al., 2004; Spais and Veloutsou, 2005);
- R&D design: product life cycle management, design to value, open innovation,
etc.;
- supply chain management: inventory management, demand
forecasting/shaping and supply planning (Trkman et al., 2010);
- production: sensor-driven operations, digital factory, etc.;
- human resource management: support to selection, knowledge management
and dissemination (Levenson, 2005; Royal and O'Donnell, 2008).

The market of BA is improving at an incredible rate. Wikibon projects the Big Data
market will reach $84B in 2026, with a CAGR of about 17% in the period 2011-2026 (Kelly,
2015), and 97% of companies with revenues of more than $100 million are already using
some form of BA (Bloomberg Businessweek Research Services, 2011). Nevertheless the
usefulness of BA in the real world is debated (Watson, 2011; Harford, 2014). Some authors,
approaching the role of BA for performance measurement, state: "Big Data can play a role in
MCSs by discovering behaviours correlated with specific goal outcomes, which would
prompt the creation of corresponding performance measures […] in general, Big Data
analyses can facilitate the discovery of important measures to be incorporated in MCSs"
(Warren et al., 2015, p. 400). Anyway they neither provide a theoretical analysis of the
potential relationship between BA and PMSs, nor a framework for facilitating the use of BA
within the control system. Analysis in terms of practice provides similar poor results: a
survey by Silvi et al., (2012) shows a general limited diffusion of BA tools connected to
performance management purposes.

2.2. Approaches to using PMS

In recent years an increasing interest in examining the "use" of PMSs has been raised.
According to Simons (1995), PMS are required to balance the fundamental organizational

4
dilemma between control and flexibility and thus, play two distinctive, but interdependent
and complementary roles: diagnostic and interactive1.
First of all, PMSs' core function is to define targets and control the capability of the
system to reaching those targets. This is the so called “diagnostic” control, that refers to the
monitoring of deliberate strategies (Mintzberg, 1978). These systems typically have three
main characteristics (Simons, 1995, p. 59): (1) the ability to measure the outputs of a process;
(2) the existence of predetermined standards against which actual results can be compared; (3)
the ability to correct deviations from standards.
Most of the discussion on the role of PMSs for strategic control has centred on their
diagnostic function. A number of PMSs frameworks have been developed, from Performance
Pyramids and Hierarchies (Dixon et al., 1990; Lynch and Cross, 1991), to Tableau de Board
(Epstein and Manzoni, 1998) and Balanced Scorecards (BSC), ( Kaplan and Norton, 1992).
All of them aiming to set up a dashboard of measures able to support day-to-day decision-
making, describe the firm's objectives, encourage coherent behaviours, and measure results.
The literature indicates the following characteristics for "strategic" PMSs: a) integration of
long-term and short-term horizons, b) integration of financial and non-financial measures, c)
combination of external and internal orientation of the measures including customer,
competitor, and supplier perspectives, d) inclusion of forward-looking perspectives (e.g.
innovation, human resource management), and e) identification of causal relationships
between the different measures and perspectives.
Available data suggest widespread adoption of comprehensive PMSs (Rigby and
Bilodeau, 2009) which has been claimed to have beneficial impact on performance (Hoque
and James, 2000; De Geuser et al., 2009). However, research also shows evidence of failure
in pursuing organizational goals (Melnyk et al., 2010; Micheli and Manzoni, 2010; Royal
Statistical Society, 2005). The most common weaknesses of PMSs observed in practice are:
(1) an inability to capture the key business success factors (Melnyk et al., 2010), (2) an
unclear impact on business performance (Franco-Santos et al., 2012) (3) a simplistic
definition of linear cause-effect relationships between financial and non-financial measures
(Ittner and Larcker, 1998; Norreklit, 2000; Silvi et al., 2015), and (4) a focus on the past and
on the short-run. Furthermore, PMSs in practice tend to be based on ex-ante assumptions,
rather than on statistical analysis intended to verify whether achieving a certain non-financial
target will result in the expected financial one. Smith and Goddarb (2002) highlight the

1
This paper concerns the concept of "use" of PMS and, in line with previous studies in the field (Abernethy and
Brownell, 1999; Bisbe and Otley, 2004; Henri, 2006; Kober et al., 2007; Su et al., 2015), focuses on two of the
levers proposed by Simons's levers of control framework (LOC). Simons (1995) conveys the idea that strategic
control derives from the interplay of four different systems (or levers), each of which has a different
informational purpose (Simons, 1995, p. 7): (1) belief systems, used to inspire and direct the search for new
opportunities; (2) boundary systems, used to set limits on opportunity-seeking behaviour; (3) diagnostic control
systems, used to motivate, monitor and reward achievement of specified goals; (4) interactive control systems,
used to stimulate organizational learning and the emergence of new ideas and strategies. However, of the four
levers, only diagnostic and interactive systems are defined as "attention patterns" and actually represent how
formal control systems are used. Conversely, boundary and belief systems are "design attributes" of formal
control and both serve the purpose of "frame[ing] the strategic domain" (Bisbe and Otley, 2004, p.711).

5
frequent subjective nature of assumptions underlying strategic planning and performance
measurements; this is responsible for performance data inadequacy.
Moreover, PMSs should play also a proactive and “creative” role. This is the activity of
the so called “interactive” control, responsible to discovering and supporting emergent
strategies (Mintzberg, 1978). In this sense, interactive controls tackle “strategic uncertainties”,
i.e. those assumptions or shocks that could "threaten or invalidate the current strategy of the
business" (Simons, 1995, p. 94). From an organizational point of view, the basic effect of
uncertainty is to limit the "ability of organizations to plan or make decisions about activities
in advance of their execution" (p. 36). Hence, uncertainty impacts the amount of information
processing necessary to perform a task (Galbraith, 1973). If there is high uncertainty, this
information is not known ahead of the task performance, which increases the need for
information processing while the task is being carried out. In these contexts, the interactive
use of PMSs is expected to increase information flows and balance commitment to the
deliberate strategy generated by the diagnostic emphasis on PMSs. More precisely,
interactive control systems are "formal information systems managers use to involve
themselves regularly and personally in the decision activities of subordinates" (Simons, 1995,
p. 95). As such, they are characterized by a high level of involvement from all organizational
members facilitating information flow. In fact, hierarchical and functional obstacles are torn
down (Abernethy and Browell, 1999; Henri, 2006) thanks to the frequent and regular
attention from operating managers at all levels of organization and from different functional
areas (Simons, 1995). As senior managers get involved in their subordinates decisions, local
information are accessed and debated. Similarly, cross-functional dialogue is stimulated as
people with different knowledge are brought together.
Research on the interactive use of PMSs is still limited though it has recently gained
momentum. The dynamism of modern business environments brings about increasing
uncertainty, and in such contexts the role of PMSs in stimulating learning and innovation
becomes pivotal. Recent contributions support this idea and show higher degrees of
"interactivity" when uncertainty increases. In particular, in the public sector, Kominis and
Dadau (2012) report a shift from diagnostic to more interactive PMSs to better manage risk
and uncertainty. Furthermore, Janke et al. (2014) conclude that when firms face crisis, the
necessary adjustments are supported by an interactive use of PMSs.
To date the discussion on interactive control has mainly concerned the "degree" of
interactivity, as explained by the presence of some distinctive features: intensive use by
senior manager, intensive use by operating manager, senior and operating managers meeting,
focus on strategic uncertainties (Naranjo-Gil and Hartmann, 2006, 2007; Bisbe et al., 2007;
Chong and Mahama, 2014). The degree of interactivity has then been investigated in relation
to innovation (Bisbe and Otley, 2004; Bisbe and Malagueno, 2009; Henri, 2006) and other
strategic capabilities (Henri, 2006) or in combination with the other levers of control
(Tuomela, 2005; Widener, 2007; Mundy, 2010). However, there is still limited understanding
of the underlying information flow and how data "generated by the system is an important
and recurring agenda" […] and "is interpreted and discussed" (Simons, 1995, p.97, Ittner and
Larcker 2005).

6
2.3. A framework for Business Performance Analytics

Moving from previous considerations, it is widely recognized the potential value that
Business Performance Analytics can add to the Performance Management Systems (Silvi et
al., 2012). With the term BPA we refer to the use of data-related analytical approach used by
managers for getting a better insight of performance dynamics and support the performance
management process.
From a diagnostic perspective, BPA can address several critical issues of PMS (Klatt et al.,
2011). First of all BPA could help managing the information overload from the perspective
of decision-makers. The amount of available data is massive and continually increasing.
Paradoxically, this information overload can represent an obstacle to the understanding of the
primary performance drivers. In this area BPA, through mathematical and statistical analysis,
could reduce the complexity of available data and lead to the understanding of the main
levers to act on for improving the final performance of the company. Secondly, BPA can help
in understanding the casual interdependences between strategic impact factors, which are
often subjectively assumed. From this point of view, BPA could have two valuable impacts.
On the one hand they could highlight the relationships between actionable variables and
strategic performance, not only in a single period of time, but also longitudinally. On the
other hand they can be of help in measuring the impact of each variable associating each one
to an expected level of variability, forecasting the expected effect and the risk associated to a
specific managerial action. Finally, BPA could support the definition of a holistic view of the
organization and of its different inputs, processes, outputs, and outcomes, connected by
critical interdependencies. As for this, BPA could improve strategic and operative planning
and measurement by providing a quantified view of the relationships between the different
inputs, activities and outputs of specific sub-processes, processes, or even entire business
units.
BPA can play a crucial role also in terms of interactive control, particularly in increasing the
ability to detect strategic uncertainties and unveil emerging strategies. The massive analysis
of data could support the understanding of unknown relationships among action and
performance variables, thus discovering new strategic patterns. Moreover, the available data
and related analytical evaluations could support the exchange of information, ideas, and
views between different departments and hierarchical levels (Simons, 1995).
Based on the previous discussion, Raffoni et al. (forthcoming) developed a practical
framework for the application of BPA. The framework aims at providing a first view of the
key aspects to be taken into consideration when applying analytical approaches to
performance management and forming a basis for further investigation. From a theoretical
standpoint, the framework is anchored to the idea that strategy control requires both single-
loop and double-loop learning (Argyris, 1977, Simons 1995) and that, for those to be
effective, a holistic view of performance management needs to be embraced (Ferreira and
Otley, 2009). Hence, the contribution of BA is discussed, drawing upon critical questions
raised with respect to their effectiveness and our understanding of their potential in
improving PMSs.

7
 The framework addresses five main issues, that at a first sight, could be considered as
subsequent steps of the process, even though in reality they should be considered
systemically, since they affect one another. The first area concerns the evaluation of the main
strategic objectives derived from an analysis of the company's strategy and business model
(issue 1). Based on this first step, the second area identifies the critical performance variables
and the specific strategic questions to be answered (issue 2). The third area focuses on data
collection (issue 3), while the fourth one concerns the design and development of the
mathematical, statistical and econometric approaches (issue 4). Finally, the last step relates to
the development of BPA's role within the whole performance management process (issue 5).
It is worth remarking that the different elements are here sequentially listed for descriptive
purposes only. We expect BPA and their mode of integration with the control process to be
more nuanced. Firstly, BPA have the potential of addressing different issues emerging from
the diagnostic and interactive use of PMSs. Secondly, their way of interacting with the PMSs
is likely to follow a different pattern.

Issue1. The analysis of the strategic objectives

In order to control strategy, understanding "which" objectives companies are required to
pursue constitute a key issue. The fact that strategic objectives may inform the PMSs should
not be interpreted as a neutral role of control with respect to strategy formulation (Anthony,
1965). In fact, strategic objectives represent the "ingredients" for any strategic view of
control (Simons, 1995; Ferreira and Otley, 2009), but when mobilized by the PMSs other
interpretations of strategy may emerge. It follows that any attempts of using BA to support
the controlling of strategy cannot disregard the comprehension of the organization's
objectives. Data and analytics can support the decision- making process but cannot replace
corporate values and the vision of what an organization intends to achieve (McAfee et al.,
2012).
Organizations pursue a number of heterogeneous targets and profit maximization is not
their unique and ultimate goal. Performance becomes therefore a complex and
multidimensional concept and cannot be described or measured on a single basis (Otley,
2008). In addition, performance is dynamic and a net of causal interdependences link
initiatives in different departments and final outcomes (Lebas and Euske, 2002).
Different kinds of performance measurement approaches have been introduced to describe
the performance model (Kaplan and Norton, 1996; Epstein et al., 2000; Ittner and Larcker,
2001). This type of maps allows the representation of the cause-and-effect relationships
between available levers and expected results (Kaplan and Norton, 2004, p. 9). The adoption
of causal maps to describe the links between performance and strategic objectives has long
been recognized as beneficial (Kaplan and Norton, 2000, 2004), as they are well suited to
simplify and facilitate the structuring of complex systems (Axelrod, 1976; Fiol and Huff,
1992). The performance models depicted through the mentioned maps are externally defined
by managers, entrepreneurs and consultants, not inferred from real data. They can help frame
the strategic environment and support the consequent application of BPA.

8
Issue 2: The identification of the critical performance variables and the specific strategic
questions to answer
The first issue provides a general view of the main performance drivers, thus framing the
strategic context of BPA. The second phase entails a more detailed approach and requires
companies to identify the strategic questions that should be addressed in order to understand
the root causes of current and future performance.
As previously discussed, diagnostic and interactive uses of PMSs deal with different issues
and help balance the need for direction and innovation which is essential to the controlling of
strategy. As for the diagnostic control, the main question is related to the specific dimensions
on which to focus the deliberated strategy of the company (Simons, 1995). Consequently, the
performance management system should be focused on the key factors that influence the
success of the intended strategy. These may be elements, competencies, and features
conceived as critical for pursuing and keeping a competitive advantage (Ferreira and Otley,
2009, p. 268).
As for interactive control, PMSs are expected to support innovation and opportunity-
seeking and thus promote strategy (re)formulation and emergent approaches. Hence, the main
questions to answer deal with which risks could shock the actual performance of the company
(Simons, 1995).
This stage of the analysis starts with an evaluation of the business world the company
competes in, and with senior management's vision for the future. While previously the pitfalls
of such an approach were put forward, here we argue that this vision still plays a crucial role
in a more evidence-based approach. The view of experienced executives, who have been
working in the industry for decades can provide a very valuable insight into critical
performance variables. In any case, this first view must be challenged and questioned by
external analysts in order to provide a wider view of the strategic context in which the
company operates and in order to derive (a few) strategic questions on which to focus the
attention of the analytical approaches.

9
Issue 3: The collection of data

The data can be collected both from internal and external sources. The most relevant
internal sources are the Management Information System, the Customer Relationship
Management System and other information systems used for specific purposes by single
departments (R&D, logistics, quality management, etc.). Relevant external sources could be
represented by social networks, blogs, open data, or specific databases purchased from
external providers.
External Big Data, characterized by high volume, variety and velocity are particularly
interesting for the potential support they can provide to interactive control. The analysis of
large amounts of unstructured data updated in real time can help in identifying strategic
patterns in progress and unveiling emerging strategies. But such kinds of data need special
attention in the data collection and cleansing processes in order to preserve the reliability of
the information (Zhang et al., 2015, Bhimani and Willcocks, 2014).
Ideally, once strategic objectives and key questions are specified, identification of the
data is needed and the related collection should logically follow. However, this sequential
logic might provide only a partial representation of reality, as a significant portion of data is
not selectively sourced. For instance, data from ERPs or Internet of Things are typically
collected along deliberate and premeditated paths reflecting specific information priorities
(Constantiou and Kallinkos, 2015) and thus fit within this argumentation. On the contrary,
most of external Big Data are generated by external users with different forms, terms and
semantic frames. In this case only an "ex-post" evaluation can transform the data into real
information able to provide a real support to the strategic decision-making process. But this
"transformation" process can affect the real informative content of the data and must be
managed carefully (Weinberger, 2007).
Considering the velocity at which this data is produced, Constantiou and Kallinkos (2015)
argued that an inductivist way of strategy-making may emerge. In other words, first acquire
data and then search for any potential use (Anderson, 2008). We believe that such an
inductive vs. deductive perspective may play an interesting role in the use of BPA for
diagnostic and interactive purposes and some tentative considerations may be drawn. As
diagnostic control is characterized by lower levels of uncertainty and consists mainly of
monitoring internally actionable variables, thus internal sources of data following a
conventional and structured architecture may mainly be relied on. Conversely, interactive
control, where focus is mainly on the monitoring of external variables, and is predominantly
inductive in nature, may find the exploitation of real-time, unstructured data more beneficial.

Issue 4: The design and development of the mathematical, statistical and econometric
approaches
The data collected must be analysed through adequate technical tools in order to extract
their full informative content and support the performance management process. In order to
have impacts on business performance, analytical insights requires a translation into
management insights and opportunities for creating value.

10
 The different techniques range from statistical and econometric approaches to tools related
to mathematics and operational research. They can be grouped into four main categories
(Gartner Report, 2012): a) descriptive analytics, aimed at describing what is happening to the
company and /or to its environment at a certain time; b) diagnostic analytics, whose final goal
is to understand the root causes of the actual performance of the company; c) predictive
analytics, able to forecast what might happen on the basis of some expectations of the
environment and the managerial action made by the company; d) prescriptive analytics, able
to provide recommendations about what should be done in order to maximize the
performance/reduce the risk.
The choice of "which" analytics is contingent to the objectives and typology of data
available to an organization and pre-conceived rules are hardly identifiable. Nevertheless, the
nature of the information exchange in the interactive and diagnostic use of BPA prompts
some provisional reflections. Diagnostic control is typically characterized by low levels of
uncertainty and the related information flow tends to follow pre-defined channels. As a
consequence, information is highly coded and characterized by a moderate level of diffusion
(Simons, 1995). In fact, it is circulated with the purpose of conserving attention and thus
limiting management's involvement. As a consequence, BPA for diagnostic control are likely
to mainly lever internal, structured data, with traditional BA and business intelligence tools
playing a major role.
Conversely, interactive control involves high level of uncertainty, thus generating the need
for higher and more flexible information processing. Data are continuously shared and
debated at different hierarchical levels with senior managements highly involved in the
process. The need for prompt and timely information implies the exchange of low-coded
information, as well as a high level of diffusion (Simons, 1995). Hence, it is expected that
external, unstructured information and the use of Big Data analytics will mainly characterize
interactive approaches.

Issue 5: The development of BPA's role within the whole performance management
process
Previous issues contributed to the definition of elements and characteristics of the BPA
approach to PMSs including: strategic factors to be monitored, key questions to be answered,
relevant data and analytical tools. How these elements are mobilized in the performance
management process is yet to be discussed and constitutes the main point of this final stage.
Figure 1 provides an illustration of such a process that, in our view, follows different patterns
according to the diagnostic or interactive use of BPA.

Figure 1 - The framework for applying BPA to performance management systems. Adapted
from Raffoni et al. (forthcoming).

11
 The diagnostic use of BPA assumes a preliminary assessment of the strategic objectives
and of the performance model. This is followed by the identification of the critical
performance variables, which is usually shaped by senior management's view of the key
factors leading to the achievement of the intended strategy. Conventionally, this stage is
followed by the selection of the key performance indicators for which targets are then set.
When introducing BPA in such a cycle, data is expected to play a role in supporting or even
challenging this top-down view of the key success factors. As a consequence, pre-defined
critical performance variables are translated into a number of questions and direct the
identification of the data needed. Then, the subsequent data analysis may take different levels
of sophistication and provide data-driven evidence of the hypothesized links among the
factors underlying the causal model of performance. Most probably, descriptive and
diagnostic BPA will play a major part at this stage. As a result, critical performance variables
may be revised and performance indicators developed accordingly. Also, BPA may influence
the setting of targets. Predictive and prescriptive BPA may be used to develop "what if"
scenarios and quantify which target level to achieve in order to obtain pre-set results. To
conclude, an effective implementation of BPA requires, in our view, the integration of such
information in the PMS. Once variables and targets are identified, measurement, reporting,
and analysis of the results will take place and drive potential corrective actions. This will then
feed and potentially adjust the strategic objectives.
As for the interactive use of BPA, it is suggested that senior management's vision of how
the business may evolve in the future still represents the starting point of the performance
management process. As a matter of fact, it is their ability to articulate this vision and identify
potential sources of strategic uncertainties that initially direct the understanding of potential
data needs. Compared to the diagnostic use of BPA, we expect the sequence strategic
objectives-data needs to be more nuanced. Strategic uncertainties mainly derive from external
changes whose monitoring is best served by timely information and flexible approaches.
Hence, external, unstructured data, including real time user-generated content, is likely to be
used for such purposes. However, the velocity and variety of data production challenges the

12
possibility of a selective data collection. The "sorting in the way out" logic is likely to occur
with senior management's expertise shaping the process. Besides, the digitization of most
data favours the sharing and communication of information at different hierarchical levels,
thus promoting debate, question and analysis that constitute the core of the interactive
function of PMS.

3. Methodology
This paper is based on a case study (Data Management Ltd -DM from now on-), a method
that offers a holistic orientation (Yin, 2008) and the possibility of understanding the potential
role of BPA. This goal requires an in-depth understanding of the company under observation
and, in particular, of the inter-relationships between the elements characterizing that complex
social system (Scapens, 1990). It also requires a combination of methods of collecting
research materials (observations, interviews, document analysis), both in quality and quantity.
Moreover, case studies are particularly useful in areas where theory is not quite developed
yet, like the case of BPAs. In conducting the case study, we adopted an interventionist
approach. The direct intervention in the case studied is particularly helpful when the target is
approaching a new phenomenon with an exploratory approach (Anderson and Widener,
2007). According to this, the researcher entered the team developing the framework in DM,
even if remaining completely independent from the company. This particular circumstance
enabled penetrative insights (Parker, 2012) and provided the opportunity to gain a better
understanding of on-going processes and observe real participants' actions and opinions
(Jönsson and Lukka, 2006). Furthermore, this approach allowed a much easier access to the
structured and unstructured data available for the company and a continuous interaction with
the managers and employees (Dumay, 2010).
The chosen interventionist approach also allowed the researchers to maintain a certain
level of flexibility in the research framework formulation, optimizing the potential of the field
study. This strategy fits in particularly with case-based fieldwork as well (Berg and Lune,
2012, pp. 22-26).

3.1 Research Site and Data Collection

Data Management is a multinational company working in the data-security industry
(databases, cloud computing, disaster recovery, etc.). Its turnover worldwide is higher than 2
billion dollars with about 10,000 employees. The turnover of the Italian branch is 130 million
euro with 260 employees. It works with both public administrations and private companies
which want to protect their data from hacking or incidents, providing both technology
(bought from the holding located in the U.K.) and services (through its local employees).
Each project involves from 5 to 25 people and it is run by a project manager (PM). Over the
2012-2015 period DM's financial performance suffered a progressive decline. Revenues
dropped from 141 to 130 million € and pre-tax profit from 8.7 to 3.9 million €. Revenues
were affected by the general financial crisis while internal operating costs experienced an
upsurge. This was accompanied by a perception that the current process configuration was no
longer suited to the organization, due to the growing complexity of managing projects.

13
 In order to support the cost management program he had in mind, the CEO of the Italian
branch contacted the author, whom he had met a few years earlier in an MBA program at
Bologna Business School. The main question was both very simple to be stated, but at the
same time very complex to be answered "What should the priorities of our company be in
order to increase financial performance?".
The research project was developed between May 2014 and October 2015. The full
commitment of the CEO played a crucial role for the success of the project. He was
enthusiastic about it and personally defended its proposition in front of the board. Other than
the CEO, the team included the author and the controller of the company, who had been
working at DM for almost ten years and his knowledge of the company was thorough
(Dumay, 2010).
Data collection was varied: semi-structured interviews, focus groups with the CEO and the
project managers, analysis of original documents and rough data, analysis of reports
developed by the company (Denzin and Lincoln, 2008; Yin, 2008).
During the first steps of the analysis, where the main aims were understanding the strategy
of the company and defining its strategic objectives, the research was mainly based on semi-
structured interviews. The CEO, the Director and Vice Director of the Project Management
Department, and the Director of the Sales Department were separately interviewed. The
interviews took place between May and June 2014 and lasted about 3 hours each. In order to
better understand the market the company was involved in, many external data were analysed:
reports regarding the general propensity of companies towards data security, financial
statements of the company and its competitors and reports about the dynamics of the whole
industry.
The focus groups, held between July and September 2014, were structured to obtain the
critical performance variables and the strategic questions to investigate through analytical
approaches. The first focus group lasted 4 hours and involved the researcher, the CEO, the
Director and Vice Director of the Project Management Department and the Director of the
Sales Department. The previous analyses and interviews were discussed in order to derive the
critical performance variables. Then, a second focus group involving the Project Management
Department (Director, Vice Director and all the 25 project managers) was created in order to
work out the strategic questions. The second focus group lasted 7 hours.
From September 2014 to February 2015, research activity was aimed at collecting the data
from different databases: the ERP system of the company (data about the ordered and billed
value of each project), the project management software (the number of hours dedicated to
each project), and some Excel files provided by the project managers (for analysing the
composition of each team working on single projects). The activity of evaluating the
reliability of data and then integrating and cleansing them was very time-consuming. On the
whole, including the time spent by the researcher, the controller, the Director, the Vice
Director of the Project Management Department and the project managers, an amount of
about 150 hours was put into this first stage.
From March 2015 to August 2015 the analytical methods were applied using SPSS
software for the econometric analysis and OSDEA software for performing the DEA analysis.

14
Some preliminary analyses and descriptive evaluations were run through Microsoft Excel.
This stage required about 85 hours.
The final part of the work, between September and October 2015 was dedicated to
presenting the result to the CEO, the Director of the Project Management Department, the
Controller and the Director of the Sales Department. Their comments were collected and a
second focus group with the CEO, the Controller and the Director of the Project Management
Department was dedicated to plan the impact of the BPA approach on the performance
management process of the company. This last stage took 2 days of work.
The data collected at each stage were coded without the adoption of any specific software.
Particular attention was dedicated to two elements emerging from the data collection: the role
of data in confirming or challenging the hypothesized steps of the framework and the critical
issues to be managed at each stage. In each phase of the analysis these points were stressed
and discussed with the Controller and the CEO in order to understand how to interpret the
information emerging from the case.

4. Results

The project started with the analysis of the Business Strategy and Performance Model of the
company. Revenues of the company come from two main businesses: hardware and
consultancy projects. Within the same global project with a customer, the company resells
hardware bought from the parent company at a given transfer price, and manages consultancy
projects in order to customize the solution. The first business unit, "hardware", is only
partially managed by the Italian branch of the company, in consideration of the fact that they
are due to buy all the hardware components from the parent company and transfer prices are
set by it. The only lever they have to increase profits in this area is to sell a higher volume of
products. As for the "consultancy" business unit, on the contrary, the Italian branch can
manage all the levers of profit: it can negotiate the prices with the customers; it can select the
engineers involved in the project and manage each single project. Furthermore, consultancy
plays a crucial role in selling the hardware, because the consultants can influence customers'
decision about which hardware to buy. Given this first strategic evaluation, attention was
focused on the "consultancy" business unit. At that time the company was collecting a huge
amount of data related to the efficiency of each consultancy project, and a detailed report of
hours spent compared to the budget was constantly made available to the Director and Vice
Director of the Project Management Department. The organization was involved in a series of
projects aimed at increasing the efficiency (read, reduce the time) of each type of projects.
Before discussing potential alternatives to the actual strategy, 25 Project Managers
(responsible for the whole process of managing the projects) and 10 Sales People
(responsible for contacting and negotiating with the customers) were interviewed in order to
better understand the factors influencing the effectiveness and efficiency of the projects. The
role of "insiders", participating in a project promoted by the company, played a fundamental
role in obtaining full access to the data and to the people to interview. This is a relevant
feature of interventionist research (Van de Ven and Johnson, 2006), that is not only
"grounded in data", but also "grounded in action", allowing the researcher to observe the

15
managers and employees when they make their decisions or do their job (Atkinson and
Shaffir, 1998).
The answers collected from the Project Managers and the Sales People were then discussed
with the CEO, the Director of the Project Managers and the Director of the Sales Department.
What emerged was that efficiency was only part of the problem. This was because at that
time efficiency was measured only by comparing the amount of hours spent on a project with
the amount forecast, without considering the cost of the consultants employed in the project.
What happened was that, whenever a project was underperforming in terms of efficiency, the
PMs selected the most experienced (and thus most expensive) consultants in order to close
the project as soon as possible, without taking into consideration the cost of that specific
intervention. As a consequence, even when the efficiency parameter, in terms of time, was
met, the real cost was often higher than the budgeted one. Secondly, the efficiency level was
not even for different kinds of projects. The Director of the PM had no instruments to
support this idea, but he was convinced that residential services, that is, when the company
just "rented" out some consultants who remained working in the ICT Department of the
customers, were much more efficient than the so called "consultancy projects", where they
worked as external consultants. Furthermore, it was thought that the projects efficiency for
the so-called "Gold Customers" (13 customers cumulatively responsible for about 45% of
total revenues) was much higher than for the "Regular Customers" (meaning, all the others).
And yet, they had no data to support this theory, but they were quite convinced that the
initiatives, for improving the efficiency of the projects, should be targeted on specific kinds
of projects and customers, due to the fact that they had special requirements. In addition, it
emerged from the focus groups that the Project Managers had the levers to manage the
efficiency of the projects, but then again, they had never measured the efficiency of the
Project Managers in order to control and enhance their performance. Furthermore, efficiency
was not the only strategic issue. The so-called "Selling rate", the hourly rate for consultancy,
obtained by the customers during the negotiation, was very different depending on different
projects and customers, representing a relevant factor in business dealing and bargaining
power of different customers. While DM had all the data about the selling rate of every
project, it had never run specific analyses in order to understand which market, customer or
service they should focus on to increase the revenues generated by maintaining the hours
spent by the consultants unchanged.
On the basis of the strategic analysis reported above, a focus group with the CEO and the
Director of the Project Managers was created to highlight critical performance variables and
strategic questions to be answered (second step of the framework). The questions are reported
hereafter:

1. How do specific kinds of projects or customers influence the selling rate of the project?
2. How do specific kinds of projects or customers influence the efficiency of the project,
calculated as the ratio between the actual hourly rate at the end of the project and the
selling rate settled during the negotiation?
3. Which project managers are more efficient in managing different kinds of projects?

16
 4. Can we use the data to predict the selling rate and the efficiency of a new project once
the typology of the project, the customer and the project manager are known?
5. Can we use the data to decide to which project manager should we give each project,
based on its characteristics?

The first three questions are descriptive and diagnostic in nature. The target is to understand
which factors influence effectiveness and efficiency of the projects and to then quantify this
effect. The fourth question is predictive in approach, using the previous analysis to forecast
the future performance of the company. Finally, the last question is prescriptive in approach,
aiming to select the right project manager (read, the most efficient one) for each specific
project.

The five questions led the data collection process (third step of the framework). The focus
of the analysis was clear (sales effectiveness and operational efficiency) and consequently
performance measures were defined (the selling rate, the ratio between the actual hourly rate
and the selling rate). At that point, a number of issues emerged. First of all, the company did
not have any data related to the winning rate, i.e. the percentage of offers accepted by the
customers compared to the total amount of presented offers. This could well be another
interesting area of BPA application. It could be used to understand which factors influenced
the winning rate and, consequently have the Sales Department focus attention and effort on
potential deals with the highest winning rate, most likely to succeed. Unfortunately, the
company did not keep any information about the offers that had not been turned into orders;
consequently any analysis on this area was substantially impossible. The only consequence of
this evaluation was that the company decided to introduce Customer Relationship
Management software for the collection of all the information with regard to the initial offer
process, allowing analyses on the winning rate for future purposes. The second issue
concerned the reliability of many data, particularly in terms of the time spent by the
employees on each project. This figure was recorded by the same employees in Excel files
and then collected by the Vice Director of the Project Management Department in order to
obtain the total amount of time spent on each project. On the basis of the interviews with the
PMs, it became clear that they often filled out the reports without the necessary attention.
Many of them considered this data recording activity a waste of time and dedicated to it only
few minutes a month. Furthermore, the collection of data through Excel files often led to
errors and unreliable final data. Only the financial data, regarding the initial budget of the
project and the final billed amount, could be considered fully reliable and able to be easily
collected.
On the whole, a relevant amount of time was spent on data collection and cleansing.
Only then, we could start applying the analytical methods for answering the critical
questions (fourth step of the framework). First, some descriptive evaluations (frequencies,
crosstabs, t-tests on means differences) were run in order to understand the data structure and
have an initial rough idea of the factors influencing the selling ratio (representing sales
effectiveness) and the actual/selling ratio (representing the efficiency in managing the
projects). Some interesting insights emerged from this analysis. It confirmed the idea that the

17
specific worth of the customer (Gold vs. Regular) had a relevant impact, while the typology
of the project (Rent vs. Consultancy) was not so relevant, particularly in influencing the
selling ratio. Consequently the suggestions provided by the focus groups were only partially
confirmed. This first analysis provided further information, highlighting huge differences in
efficiency related to the size of the project (“Small” if lower than 25,000 €; “Big” if higher).
On the basis of the points emerging as relevant in the first descriptive analyses, we
conducted a cluster analysis that highlighted four very different clusters: “Big” Projects for
"Gold" Customers (BPGC), "Small" Projects for "Regular" Customers (SPRC), "Small"
Projects for "Gold" Customers (SPGC) and "Big" Projects for "Regular" Customers (BPRC).

Figure 2: The features of the 4 clusters emerging from the clustering procedures

The characteristics of the four clusters raised interesting managerial issues. First, the
selling rates of the projects with Gold Customers were considerably lower than those with
Regular Customers, particularly for Big Projects. While the attention of the Sales Department
was highly absorbed by Gold Customers, the Regular ones showed a relevant potential. Big
Projects with Gold Customers, on the contrary, show much higher efficiency if compared
with other projects, especially smaller ones. This outcome suggests that the most efficient
project managers and consultants were employed for the most important projects (Big jobs
involving Gold customers), while less attention was dedicated to small projects involving
regular customers. In order to confirm this hypothesis, we decided to further investigate the
efficiency of the different Project Managers, because as suggested by the focus groups, they
play a crucial role in influencing efficiency.
For this purpose, we decided to use Data Envelopment Analysis (Banker et al., 1984), a
non parametric mathematical model for the estimate of the efficiency of given decision-
making units (DMU), highlighting the inputs and the outputs of the production process. The
amount of hours dedicated to each project was considered as the input and the expected hours
and the number of projects managed were defined as the outputs.

18
 Figure 3: The efficiency of PMs obtained through the Data Envelopment Analysis.

It is easy to see that the efficiency of PMs is considerably heterogeneous, with efficiency
scores ranging from 0.51 to 1. This analysis might as well lead to performance measures
assigned to each PM and to educational programs able to increase the performance of the less
efficient PMS.
Finally, a DEA was run for each cluster of projects in order to obtain the efficiency of each
PM for each cluster of projects.
This analysis shows how the efficiency of many PMs was very dissimilar from cluster to
cluster. Some PMs were very efficient in managing small projects, while others were much
more productive in big projects, some were able to manage projects with Gold Customers and
others were much more efficient with Regular Customers. Less than 40% of PMs showed low
variances between different kinds of projects.
This result was very useful in answering the last two critical questions related to the
predictive and prescriptive use of BPA in DM.
Once we know the specific cluster the project belongs to, and the specific PM given to that
project, we can then predict, with a known level of uncertainty, the expected selling rate and
the related efficiency, thus forecasting the expected output of the project. And, as a
consequence of the last analysis, it is possible to select the most efficient available PM for a
specific project. Even more so, a PM could be moved from one project to a new one, if the
consequence would be an increase in the overall outcome of the company. The analysis
provided to diagnostic control a clear, tested and objective new method to measure the
performance of PMs. As for interactive control, it represents the possibility to select at each
moment the most efficient and suitable PM, even removing a PM from a project he/she is
employed in, if that means an increase of the company overall performance.
The CEO was very excited by the results of the analytical applications and decided to
immediately introduce them into the performance management system of the company.

19
First of all, as mentioned earlier, he decided to introduce a CRM system in order to
understand the winning rate of different kinds of projects and the factors influencing the
winning rate. Secondly, each project, once having obtained the order by the customer, was
classified on the basis of its cluster and hence its expected output was forecast, and the most
efficient PM, appropriate to the project, was selected. The performance of each PM was
measured and rewarded on the basis of his DEA and a global performance measure for the
company was defined as follows:

Global Consultancy Performance (GCP) = Winning Rate%*Selling rate%*Efficiency%

Where:
- Winning Rate% (WR) = amount of winning offers on total amount of offers;
- Selling rate% (SR) = Selling Rate of each project compared to the highest Selling
Rate of its Cluster;
- Efficiency% (E) = ratio between the actual hourly rate at the end of the project and the
selling rate.

The GCP takes into consideration the capability of acquiring new orders (through the WR),
the capability to sell services at high prices (through the SR) and efficiency (through E).
The CEO was very satisfied by the result, and commented: "We already knew some of the
things which emerged from the analysis, but we ignored some other results, and above all, we
couldn't express our beliefs with numbers so extensively. The potential usefulness of BPA is
twofold in my opinion. First, they help in making decisions and second they could be the
basis of the new PMS. Now we have to turn this knowledge into plans, targets, measures and
controls”.

5. Discussion and conclusions

This paper explores the role of BA for performance measurement purposes. Building on
Simons's notion of diagnostic and interactive use of control (1995), it supports the integration
of BA within PMSs for strategic control. It does so by testing an operational framework
(Raffoni et al. forthcoming) on a specific case study and presenting a number of issues to be
taken into consideration when designing and implementing BPA. The framework was tested
in a data-security company by adopting an interventionist approach. The results obtained
support the relevance of the five steps of the original framework and they provide insights for
future developments. First, the case study contributed to a better comprehension of how BPA
can support the identification of critical performance variables and related interdependences.
When DM was approached, several of the issues often associated in the implementation of
PMS were observed (Klatt et al., 2011):
- information overload - within the DM project, data were partial (no data about the
winning rate were available), quite sparse and only partially reliable. Commercial data
were collected by the Sales Department, accounting information by the
Administrative Department and operational data by the Program Management

20
 Department. This condition is advocated by the literature as one of the most relevant
limitations for an effective use of data for PMSs purposes (Nudurupati et al., 2011);
- the absence of measured casual interdependences - some causal relationships were
hypothesized, but DM had never performed any analysis to test such hypotheses. The
poor definition of causal effect linkages is considered a major issue in PMSs
implementation (Brignall, 2002), yet it is considered a critical element of strategic
PMSs (Chenhall, 2005; Gimbert et al., 2010)
- the lack of a holistic view of the organization - Data were fragmented and separately
collected to measure the performance of each Department and not brought together to
allow comprehension of the whole performance model. Such a compartmental
approach prompted excessive attention to cost reduction within each separate process.
This compromised the opportunity to focus on organizational effectiveness and take
more comprehensive initiatives.

The application of the BPA framework at DM provided some interesting results and also
unveiled the criticality of a number of aspects only partially highlighted by the framework.
Firstly, the initial discussion on DM's strategic objectives provided the company with a
clearer view of critical performance variables. Some critical aspects were revealed (the
winning rate of new offers, the selling rate), while efficiency, at the time the main strategic
objective of the company, was viewed in a different light (new measures, different views for
different kind of projects).
Secondly, the framework allowed the overcoming of a relevant weakness of BA
implementation: the lack of connectivity to the key success factors of the organization. The
clear evaluation of the strategic priorities of DM supported the definition of the critical
strategic questions and the subsequent identification of the relevant data for analysing
performance variables and tackling strategic uncertainties. These preliminary considerations
avoided a huge amount of myopic data crunching (Grant et al., 2006). However, different
patterns in the relationship between data and strategy can be hypothesized with respect to
diagnostic and interactive control. A more strategy-based analysis, focused on critical
performance variables within deliberate strategies is likely to be a better fit to diagnostic
control; conversely, a more data-driven analysis, less fastened by ex-ante theories could be
more useful for understanding strategic questions, but only partially deserves further
investigation. In DM the analysis revealed some aspects previously not considered, such as
the impact of the dimension of the project on its efficiency, while refuted some others
considered relevant by the managers (the impact of the specific typology of project).
Consequently, BPA play two different roles in terms of the connection between strategy and
performance measures: from a diagnostic point of view they reveal and quantify the
relationship between strategic variables and performance within intended strategies, while,
from an interactive point of view, they help capture new relationships among variables and
reveal potential emergent strategies.
Thirdly, in the case of DM, the analyses performed were able to support all the different
aims of the analytical performance evaluation (Gartner Report, 2012): descriptive (through
frequencies and crosstabs), diagnostic (through cluster analysis), predictive and prescriptive

21
(the two DEA models). The four types of analytics reflect different degrees of complexity
and, as an organization moves along the descriptive-prescriptive spectrum, the acquisition
and development of additional competencies is required. This is likely to trigger the
emergence of new organizational positions, such as the data scientist (Davenport and Patil,
2012), but also significant changes in the roles and competencies of more traditional figures
(e.g. controllers, managers, CFOs). As a consequence, there are significant implications for
both post graduate and continuing executive education.
The case study also provided some insights into a number of critical aspects that the
introduction of evidence-driven approaches may bring about. As already mentioned,
engaging with a more data-driven strategic control is likely to generate cultural and
competency issues. In fact, this approach requires different mind-sets and skills. For instance,
in the case of DM, at the beginning the Director and the Vice Director of the Project
Management Department did not appreciate the new BPA approach. They were used to being
evaluated only on the number of hours spent on each project compared to the budgeted ones.
Utilizing measurement through the Data Envelopment Approach, a mathematical tool they
were not aware of, was shocking to them. A significant amount of time was spent explaining
the "whys" and the "how" of the new approach. Theory suggests that the use of a data-driven
approach produces a more "objective" and selective decision-making; but, on the other hand,
managers might feel deprived of part of their decision-making power. Thus, staff training and
skills development may turn out to be as important as data and statistics (Orlikowski 2000,
Nudurupati et al. 2011).
Another critical point for the framework is the availability of a huge amount of reliable
data. As with smaller datasets, larger ones may present errors, biases, and missing values
(Bhimani and Willcocks, 2014). Without reliable and detailed data a BPA approach is
completely useless.
In sum, based on previous considerations, we believe that our research project delivers
interesting contributions, both in theory and in practice.
First, by focusing on the role of evidence-based decision-making, this study expands upon
previous research that has explored the role of data analysis in the context of performance
management (Ittner and Larcker, 2005). By considering both the diagnostic and interactive
function of BPA, it also addresses the call for a better development of theory on the use of
PMSs (Ferreira and Otley, 2009). In particular, it helps to operationalize the ambiguous
notion of interactive control (Tessier and Otley, 2012) by clarifying how BPA can be
employed to analyse strategic uncertainties and focus diagnostic controls on the most critical
performance variables. This kind of analysis also allows the evaluation of the risk associated
with a specific performance. Indeed, the use of statistical tools permits the computation of the
estimated level of error and variance within the performance model.
Secondly, it tests a framework for applying BPA. A series of issues are identified and then
explored in relation to the use of PMSs and different patterns in the relation data-strategic
control emerge. The strategy-driven view of data is challenged when BPA are used
interactively (Constantiou and Kallinikos, 2015) and the main target becomes that of
providing timely information for a prompt reaction. Also, given the distinctive features of
diagnostic and interactive BPA, different data and analytical methods appear to be relevant.

22
When BPA are adopted for diagnostic control they should be focused on clear strategic
targets and primarily applied to well-structured datasets, through BI and BA. Conversely,
when implemented interactively in order to explore emergent strategies, BPA can often be
based on Big Data.
Third, by adopting an interventionist approach, we also contribute to an emerging stream
of literature in the management accounting field, and address the call for further exploration
of the potential for this approach to practice-relevant management accounting theory (Lukka
and Suomala, 2014). In this sense, our work provides a detailed account of the interventionist
process and highlights the benefits and challenges of the "insider" role played by the
researcher.
This paper is based on a single case study and the scope is thus limited. Therefore, a
statistical generalization of our conclusions is not possible. However, a complete explanation
of the framework and a detailed narrative of the research process has been provided, in order
that it be retained, used and tested in future additional cases. Our focus was on the production
of an original solution for DM, based on previous theories and cases, adapted to its actual
situation, but potentially generalized and diffused to other cases, in order to make a
theoretical contribution. In particular, the relationship between data and uses of control calls
for further empirical investigation. More dynamic organizations, as well as companies where
external data play a more significant role, could represent interesting research contexts for
further studies.

References

Abernethy, M.A. and Brownell, P. (1999), “The role of budgets in organizations facing
strategic change: an exploratory study”, Accounting, Organizations and Society, Vol. 24, pp.
189-204.

Anderson, C. (2008), The end of theory: The data deluge makes the scientific method
obsolete, Wired Magazine, Vol. 16 No. 7, available at
http://archive.wired.com/science/discoveries/magazine/16-07/pb_theory, last access 30th
September 2015.

Anderson, S.W. and Widener, S.K. (2007), “Doing Quantitative Field Research in
Management Accounting”, in Chapman, C.S., Hopwood, A.G. and Shields, M.D. (Eds.),
Handbook of Management Accounting Research, Elsevier Ltd., Amsterdam, pp. 319-341.

Anthony, R.N. (1965), Planning Control Systems. A Framework for Analysis, Harvard
University Press, Boston, MA.

Argyris, C. (1977), “Organizational learning and management information

systems”, Accounting, Organizations and Society, Vol. 2 No. 2, pp. 113-123.

Atkinson, A.A. and Shaffir, W. (1998), “Standards for field research in management
accounting”, Journal of Management Accounting Research, Vol. 10, pp. 41-68.

23
Axelrod, R. (Ed.) (1976), Structure of decision: the cognitive maps of political elites,
Princeton University Press, Princeton.

Banker R., Charnes A. and Cooper W. (1984), Some models for estimating technical and
scale inefficiencies in Data Envelopment Analysis. Management Science;30(9):1078–92.

Berg, B.L. and Lune, H. (2012), Qualitative Research Methods for the Social Sciences, eight
ed. Pearson, Edinburgh.

Berry, A.J., Coad, A.F., Harris, E.P., Otley, D.T. and Stringer, C. (2009), “Emerging themes
in management control: A review of recent literature”, The British Accounting Review, Vol.
41 No. 1, pp. 2-20.

Bhimani, A. and Willcocks, L. (2014), “Digitisation, “Big Data” and the transformation of
accounting information”, Accounting and Business Research, Vol. 44 No. 4, pp. 469-490.

Bisbe, J. and Malagueño, R. (2009), “The choice of interactive control systems under
different innovation management modes”, European Accounting Review, Vol. 18 No. 2, pp.
371-405.

Bisbe, J. and Otley, D. (2004), “The effects of the interactive use of management control
systems on product innovation”, Accounting, Organizations and Society, Vol. 29 No. 8, pp.
709-737.

Bisbe, J., Batista-Foguet, J.M. and Chenhall, R.H. (2007), “Defining management accounting
constructs: a methodological note on the risks of conceptual misspecification”,

Bloomberg Businessweek Research Services (2011), The Current State of Business Analytics,
available at http://www.sas.com/resources/asset/busanalyticsstudy_wp_08232011.pdf, last
access 21st September 2015.

Brignall, S. (2002), “The Unbalanced Scorecard: A Social and Environmental Critique” in

Neely, A. and Walters, R.A. (Eds.), Performance Measurement and Management: Research
and Action, Performance Measurement Association, Boston, pp. 85-92.

Chapman, C.S. and Kihn, L.A. (2009), “Information system integration, enabling control and
performance”, Accounting, Organisations and Society, Vol. 34 No. 2, pp. 151-169.

Chartered Institute of Management Accountants -CIMA (2014), Big Data. Readying business
for the big data revolution, available at
http://www.cgma.org/Resources/Reports/DownloadableDocuments/CGMA-briefing-big-
data.pdf

Chenhall, R.H. (2005), “Integrative strategic performance measurement systems, strategic

alignment of manufacturing, learning and strategic outcomes: an exploratory
study”, Accounting, Organizations and Society, Vol. 30 No. 5, pp. 395-422.

24
Chong, K.M. and Mahama, H. (2014), “The impact of interactive and diagnostic uses of
budgets on team effectiveness”, Management Accounting Research, Vol. 25 No. 3, pp. 206-
222.

Constantiou, I.D. and Kallinikos, J. (2015), “New games, new rules: big data and the
changing context of strategy”, Journal of Information Technology, Vol. 30 No. 1, pp. 44-57.

Davenport, T.H. and Patil, D.J. (2012), “Data scientist”, Harvard Business Review, Vol. 90
No. 10, pp. 70-76.

Davenport, T.H., Harris, J.G. and Morison, R. (2010), Analytics at Work: Smarter Decisions.
Better Results, Harvard Business Press, Boston.

De Geuser, F., Mooraj, S. and Oyon, D. (2009), “Does the balanced scorecard add value?
Empirical evidence on its effect on performance”, European Accounting Review, Vol. 18 No.
1, pp. 93-122.

Denzin, N.K. and Lincoln, Y.S. (2008), Collecting and interpreting qualitative materials, Vol.
3, Sage, London.

Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1990), The New Performance Challenge.
Measuring Operations for World Class Competition, Business One Irwin, Illinois.

Dumay, J.C. (2010), “A critical reflective discourse of an interventionist research project”,

Qualitative Research in Accounting & Management, Vol. 7 No. 1, pp. 46-70.

Epstein, M.J., Kumar, P. and Westbrook, R.A. (2000), “The drivers of customer and
corporate profitability: modelling, measuring, and managing the causal
relationships”, Advances in Management Accounting, Vol. 9, pp. 43-72.

Epstein, M. and Manzoni, J.F. (1998), “Implementing Corporate Strategy: From Tableaux de
Bord to Balanced Scorecards”, European Management Journal, Vol. 16 No. 2, pp. 190-203.

Evans, J.R. (2013), Business Analytics: Methods, Models, and Decisions, Prentice- Hall,
Englewood Cliffs, NJ.

Ferreira, A. and Otley, D. (2009), “The design and use of performance management systems:
An extended framework for analysis”, Management Accounting Research, Vol. 20 No. 4, pp.
263-282.

Fiol, C.M. and Huff, A.S. (1992), “Maps for managers: where are we? Where do we go from
here?”, Journal of Management Studies, Vol. 29 No. 3, pp. 267-285.

Franco-Santos, M., Lucianetti, L. and Bourne, M. (2012), “Contemporary performance

measurement systems: A review of their consequences and a framework for
research”, Management Accounting Research, Vol. 23 No. 2, pp. 79-119.

Galbraith, J.R. (1973), Designing complex organizations, Addison-Wesley Publishing

Company, Mass.
25
Galeotti, M., Garzella, S., Fiorentino, R., & Della Corte, G. (2016). The Strategic Intelligence
implications for Information Systems. Management Control,1, pp. 105-123.

Gartner Report (2012), Forecast: Enterprise software markets. Worldwide, 2011-2016, 4Q12
Update, available at http://www.gartner.com/resId= 2054422, last access 18th March 2013.

Gimbert, X., Bisbe, J., Mendoza, X., 2010. The Role of Performance Measurement Systems
in Strategy Formulation Processes. Long Range Planning, 43, 477-497.

Grant, D., Hall, R., Wailes, N. and Wright, C. (2006), “The false promise of technological
determinism: the case of enterprise resource planning systems”, New Technology, Work and
Employment, Vol. 21 No. 1, pp. 2-15.

Harford, T. (2014), “Big data: A big mistake?”, Significance, Vol. 11 No. 5, pp. 14-19.

Henri, J.F. (2006), “Management control systems and strategy: a resource-based perspective”,
Accounting, Organizations and Society, Vol. 31 No. 6, pp. 529-558.

Holsapple, C.W. and Whinston, A. (1996), Decision Support Systems: A Knowledge-Based

Approach, West, St. Paul, MN.

Holsapple, C.W., Lee-Post, A. and Pakath, R. (2014), “A unified foundation for business
analytics”, Decision Support Systems, Vol. 64, pp. 130-141.

Hoque, Z. and James, W. (2000), “Linking balanced scorecard measures to size and market
factors: impact on organizational performance”, Journal of Management Accounting
Research, Vol. 12 No. 1, pp. 1-17.

Hyvönen, J. (2007), “Strategy, performance measurement techniques and information

technology of the firm and their links to organisational performance”, Management
Accounting Research, Vol. 18 No. 3, pp. 343-366.

Ittner, C.D. and Larcker, D.F. (2005), “Moving from strategic measurement to strategic data
analysis”, in Chapman, C.S. (Ed.), Controlling strategy: Management, accounting, and
performance measurement, Oxford University Press, New York, pp. 86-105.

Ittner, C.D. and Larcker, D.F. (2001), “Assessing empirical research in managerial
accounting: a value-based management perspective”, Journal of Accounting and Economics,
Vol. 32 No. 1-3, pp. 349-410.

Ittner, C.D. and Larcker D.F. (1998), “Are Non Financial Measures Leading Indicators of
Financial Performance? An Analysis of Customer Satisfaction”, Journal of Accounting
Research, Vol. 36 Supplement, pp. 1-35.

Janke, R., Mahlendorf, M.D. and Weber, J. (2014), “An exploratory study of the reciprocal
relationship between interactive use of management control systems and perception of
negative external crisis effects”, Management Accounting Research, Vol. 25 No. 4, pp. 251-
270.

26
Jönsson, S. and Lukka, K. (2006), “There and Back Again: Doing Interventionist Research in
Management Accounting”, in Chapman, C.S., Hopwood, A.G. and Shields, M.D. (Eds.),
Handbook of Management Accounting Research, Elsevier Ltd., Amsterdam, pp. 373-397.

Kaplan, R.S. and Norton, D.P. (2004), “How strategy maps frame an organization’s
objectives”, Financial Executive, Vol. 20 No. 2, pp. 40-45.

Kaplan, R.S. and Norton, D.P. (2000), Strategy-focused Organization: How Balanced
Scorecard Companies Thrive in the New Business Environment, Harvard Business School
Press, Boston, MA.

Kaplan, R.S. and Norton, D.P. (1996), The Balanced Scorecard: translating strategy into
action, Harvard Business School Press, Boston.

Kaplan, R.S. and Norton, D.P. (1992), “The Balanced Scorecard – Measures that Drive
Performance”, Harvard Business Review, Vol. 70 No. 1, pp. 71-79.

Kasanen, E., Lukka, K. and Siitonen, A. (1993), “The constructive approach in management
accounting research”, Journal of Management Accounting Research, Vol. 5 (Fall), pp. 241-
264.

Kelly J. (2015), Big Data: From Promise to Reality, available at http://wikibon.com/big-data-

from-promise-to-reality/, last access 21st September 2015.

Klatt, T., Schläfke, M., and Möller, K. (2011), “Integrating business analytics into strategic
planning for better performance”, Journal of Business Strategy, Vol. 32 No. 6, pp. 30-39.

Kober, R., Ng, J. and Paul, B.J. (2007), “The interrelationship between management control
mechanisms and strategy”, Management Accounting Research, Vol. 18 No. 4, pp. 425-452.

Kominis, G. and Dudau, A.I. (2012), “Time for interactive control systems in the public
sector? The case of the Every Child Matters policy change in England”, Management
Accounting Research, Vol. 23 No. 2, pp. 142-155.

Lebas, M. and Euske, K. (2002), “A conceptual and operational delineation of performance”,

in Neely, A. (Ed.), Business performance measurement. Theory and practice, Cambridge
University Press, Cambridge, pp. 65-80.

Levenson, A. (2005), “Harnessing the power of HR analytics”, Strategic HR Review, Vol. 4

No. 3, pp. 28-31.

Lukka, K. and Suomala, P. (2014), “Relevant interventionist research: balancing three

intellectual virtues”, Accounting and Business Research, Vol. 44 No. 2, pp. 204-220.

Lynch, R.L. and Cross, K.F. (1991), Measure Up! Yardstick For Continuous Improvement,
Blackwell, Cambridge, MA.

Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C. and Byers, A.H.
(2011), Big data: The next frontier for innovation, competition, and productivity, McKinsey
27
Global Institute, available at
http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_inn
ovation, last access 13th July 2015.

Maule, R.W., Schacher, G., Gallup, S.P. and McClain, B. (2004), “Applied knowledge
analytics for military experimentation”, in Proceedings of IEEE International Conference on
Information Reuse and Integration, pp. 91-96.

McAfee, A., Brynjolfsson, E., Davenport, T.H., Patil, D. and Barton, D. (2012), “Big data.
The management revolution”, Harvard Business Review, Vol. 90 No. 10, pp. 61-67.

Melnyk, S.A., Bititci, U., Platts, K., Tobias, J. and Andersen, B. (2014), “Is performance
measurement and management fit for the future?”, Management Accounting Research,
Vol. 25 No. 2, pp. 173-186.

Melnyk, S.A., Hanson, G.D. and Calantone, R.J. (2010), “Hitting the Target… But Missing
the Point: Resolving the Paradox of Strategic Transition”, Long Range Planning, Vol. 43 No.
4, pp. 555-574.

Micheli, P. and Manzoni, J.F. (2010), “Strategic Performance Measurement: Benefits,

Limitations and Paradoxes”, Long Range Planning, Vol. 43 No. 4, pp. 465-476.

Mintzberg, H. (1978), “Patterns of strategy formulation”, Management Science, Vol. 24 No.

9, pp. 934-948.

Moss, L.T. and Atre, S. (2003), Business intelligence roadmap: the complete project lifecycle
for decision-support applications, Addison-Wesley Professional.

Mundy, J. (2010), “Creating dynamic tensions through a balanced use of management control
systems”, Accounting, Organizations and Society, Vol. 35 No. 5, pp. 499-523.

Naranjo-Gil, D. and Hartmann, F. (2007), “Management accounting systems, top

management team heterogeneity and strategic change”, Accounting, Organizations and
Society, Vol. 32 No. 7, pp. 735-756.

Naranjo-Gil, D. and Hartmann, F. (2006), “How top management teams use management
accounting systems to implement strategy”, Journal of Management Accounting
Research, Vol. 18 No. 1, pp. 21-53.

Norreklit, H. (2000), “The Balance on the Balanced Scorecard. A critical analysis of Its
Assumptions”, Management Accounting Research, Vol. 11 No. 1, pp. 65-88.

Nudurupaty, S.S., Bititci, U.S., Kumar, V. and Chan, F.T.S. (2011), “State of the art literature
review on performance measurement”, Computers and Industrial Engineering, Vol. 60 No. 2,
pp. 279-290.

Orlikowski, W. J., 2000. Managing use not technology: a view from the trenches. Mastering
Information Management. London: Prentice-Hall.

28
Otley, D. (2008), “Did Kaplan and Johnson get it right?”, Accounting, Auditing &
Accountability Journal, Vol. 21 No. 8, pp. 229-239.

Parker, L.D. (2012), “Qualitative Management Accounting Research: Assessing deliverables

and relevance”, Critical Perspectives on Accounting, Vol. 23 No. 1, pp. 54-70.

Power, D.J. and Sharda, R. (2009), “Decision support systems”, in Nof, S.Y. (Ed), Handbook
of Automation, Part I, pp. 1539-1548.

Raffoni A., Bartolini M., Silvi R., Visani F. (forthcoming), “Business Performance Analytics:
Exploring the Potential for Performance Management Systems”, Production Planning &
Control.

Rigby, D. and Bilodeau, B. (2009), Management tools and trends 2009, Bain & Company.

Royal, C. and O'Donnell, L. (2008), “Emerging human capital analytics for investment
processes”, Journal of Intellectual Capital, Vol. 9 No. 3, pp. 367-379.

Royal Statistical Society (2005), Performance indicators: good, bad and ugly, available at
http://www.rss.org.uk/PDF/PerformanceMonitoring.pdf, last access 10th March 2010.

Scapens, R.W. (1990), “Researching Management Accounting Practice: the Role of Case
Study Methods”, British Accounting Review, Vol. 22 No. 3, pp. 259-281.

Schläfke, M., Silvi, R. and Möller, K. (2012), “A framework for business analytics in
performance management”, International Journal of Productivity and Performance
Management, Vol. 62 No. 1, pp. 110-122.

Schreyögg, G. and Steinmann, H. (1987), “Strategic Control: A New Perspective”, The

Academy of Management Review, Vol. 12 No. 1, pp. 91-103.Sharda, R., Aronson, J. E. and
King, D. (2008), Business intelligence: A managerial approach, Pearson Prentice Hall, Upper
Saddle River.

Silvi, R., Bartolini, M., Raffoni, A. and Visani, F. (2015), “The practice of strategic
performance measurement systems: Models, drivers and information
effectiveness”, International Journal of Productivity and Performance Management, Vol. 64
No. 2, pp. 194-227.

Silvi, R., Bartolini, M., Raffoni, A., and Visani, F. (2011). Business Performance Analytics:
il valore emergente nei sistemi di management accounting. Management Control, Vol. 3,
pp.17-37.

Silvi, R., Bartolini, M., Raffoni, A. and Visani, F. (2012), “Business Performance Analytics:
level of adoption and support provided to Performance Measurement Systems”, Management
Control, Vol. 3 Special Issue, pp. 117-142.

Simons, R. (1995), Levers of control. How Managers Use Innovative Control Systems to
Drive Strategic Renewal, Harvard Business School Press, Boston.

29
 Smith, P.C. and Goddard, M. (2002), “Performance management and operational research: a
marriage made in heaven?”, Journal of the Operational Research Society, Vol. 53 No. 3, pp.
247-255.

Spais, G. and Veloutsou, C. (2005), “Marketing analytics: Managing incomplete information

in consumer markets and the contribution of mathematics to the accountability of marketing
decision”, South European Review of Business Finance and Accounting, Vol. 3 No. 1, pp.
127-150.

Stubbs, E. (2011), The value of Business Analytics, Vol, 43, John Wiley & Sons Inc., New
Jersey.

Su, S., Baird, K. and Schoch, H. (2015), “The moderating effect of organisational life cycle
stages on the association between the interactive and diagnostic approaches to using controls
with organisational performance”, Management Accounting Research, Vol. 26, pp. 40-53.

Tessier, S. and Otley, D. (2012), “A conceptual development of Simons’ Levers of Control

framework”, Management Accounting Research, Vol. 23 No. 3, pp. 171-185.

Trkman, P., McCormack, K., Valadares de Oliveira, M.P. and Bronzo Ladeira, M. (2010),
“The impact of business analytics on supply chain performance”, Decision Support Systems,
doi:10.1016/j.dss.2010.03.007.

Tuomela, T.S. (2005), “The interplay of different levers of control: a case study of
introducing a new performance measurement system”, Management Accounting Research,
Vol. 16 No. 3, pp. 293-320.

Warren, J.D., Moffitt, K.C. and Byrnes, P. (2015), “How Big Data will change accounting”,
Accounting Horizons, Vol. 29 No. 2, pp. 397-407.

Watson, H.J. (2011), “Business analytics insight: hype or here to stay?”, Business
Intelligence Journal, Vol. 16 No. 1, pp. 4-8.

Weinberger, D. (2007), Everything is miscellaneous: The power of the new digital disorder,
Macmillan.

Widener, S.K. (2007), “An empirical analysis of the levers of control framework”,
Accounting, Organizations and Society, Vol. 32 No. 7/8, pp. 757-788.

Yin, R.K. (2008), Case study research: design and methods, Sage Publications, Beverly Hills,
CA.

Zhang, J., Yang, X. and Appelbaum, D. (2015), « Toward effective Big Data analysis in
continuous auditing”, Accounting Horizons, Vol. 29 No. 2, pp. 469-476.

30

View publication stats