SSRN Id3941961

Empirical research frameworks in a changing world:
The case of audit data analytics
Klaus Ruhnke
Freie Universitaet Berlin, School of Business and Economics

Department of Finance, Accounting and Taxation
Thielallee 73, D-14195 Berlin, Germany
klaus.ruhnke@fu-berlin.de
https://orcid.org/0000-0003-3296-9409
June 2, 2022
Electronic copy available at: https://ssrn.com/abstract=3941961

Empirical research frameworks in a changing world:
The case of audit data analytics
Abstract
Advances in data analytics is fundamentally changing the field of auditing. The primary
objective of this paper is to develop a holistic, multidimensional framework to better
understand the dynamics of the diffusion of audit data analytics (ADA). The framework we
developed covers categories related to the diffusion of ADA (the technology, the actors using
it, the external rules that regulate it) and the actors reproducing the rules and technologies
(focusing on the audit process, cognitive processes). We analyze the key findings from 40
empirical studies within and across the interrelated categories of the framework. The
findings emphasize that a better understanding of auditors’ judgment and decision making
(cognitive processes) is fundamental for a study of the diffusion of ADA. Primary drivers of
fundamental changes include audit quality, audit efficiency and the expectation gap. Our
results suggest that ADA competencies should be a professional priority. In addition,
conceptual issues (the audit risk model, the concepts of audit evidence, audit assertions) and
materiality considerations need to be resolved. A solution of these problems is a prerequisite
for in-depth changes to audit standards, which in turn will also open the door to the diffusion
of ADA.
Keywords: audit; audit evidence; big data; data analytics; empirical research, framework.
JEL Classifications: D91, M42, O30.

1 Introduction
Recent changes in data analytics have begun to affect the functioning of audit firms, audit
processes, and the highly regulated environment in which they operate. The external audit
of financial statements is undergoing a data revolution and the future changes are likely to
be far more profound than the influence that is observable in the current audit practice.
Therefore, it is not surprising that data analytics is often mentioned as a promising stream
for future innovative audit research (Köhler & Quick, 2018; Walker & Brown-Liburd, 2019).
However, for various reasons, it is difficult to tackle this multi-faceted research field.
Data analytics is a broad term, often defined as “the science and art of discovering and
analysing patterns, deviations and inconsistencies, and extracting other useful information
in the data underlying or related to the subject matter of an audit through analysis, modelling
and visualization for the purpose of planning or performing the audit” (International
Auditing and Assurance Standards Board (IAASB), Data Analytics Working Group
(DAWG), 2016, 7). In general, data analytics refers to data sources and the techniques of
data analysis. The use of data analytics as part of an audit is not new. However, more client
data and external data sources are now used. Big Data includes data of high volume, high
variety, and high velocity that can be processed electronically for enhanced insight and
decision-making (Gartner, 2021). This progression fueled the usage of advanced data
analysis techniques (e.g., process mining, text mining, visualization, and predictive
analytics).
Traditional data analysis still dominates current audit practices (Chartered Professional
Accountants (CPA) Canada, 2017, 2019; Eilifsen et al., 2020). Traditional, substantive
analytical audit procedures require the development of expectations (mostly referring to
aggregated data), which then have to be compared with the data at hand (confirmatory data
analysis). Advanced data analytics techniques are often explorative, to discover anomalies
or correlations in large disaggregated data sets, which are then examined or interpreted with
regard to their audit relevance (Alles & Gray, 2016; Liu, 2014). These exploratory and
confirmatory data-analysis techniques do not compete, but are used jointly and iteratively
(Byrnes et al., 2015; Richins et al., 2017). However, there is no clear dividing line between
traditional and advanced data analytics. With this in mind, this paper focuses on recent

developments in this area and preferably uses the term ‘audit data analytics’ (ADA). 1 In the
following, ADA will only be considered in the context of a financial statement audit.
Audit-related research from recent ADA developments is mostly conceptual, often
describing ADA characteristics via various examples, discussing its current development
and potential obstacles, and pointing out areas requiring further research (Alles & Gray,
2016; Appelbaum et al., 2017; Cao et al., 2015; Jans et al., 2014; Vasarhelyi et al., 2015).
Recent empirical studies are often survey- or interview-based. These exploratory studies
primarily focus on the emergence of recent ADA techniques in auditing (Austin et al., 2021;
Eilifsen et al., 2020; Salijeni et al., 2019; Walker & Brown-Liburd, 2019). In addition,
research based on experiments, archival studies, and case studies, shed light on isolated,
narrowly defined issues. Examples include the timing of the use of data visualizations in the
audit process (Rose et al., 2017) or the use of predictive ADA techniques for fraud detection
(Perols et al., 2017). A barrier to further research is the lack of research data, since big audit
firms develop new ADA applications to gain a competitive advantage. Moreover, some
applications are not yet mature.
In this context, this paper brings together a variety of perspectives to develop an overall
view about the diffusion of ADA. Our multidimensional, holistic framework aims to better
understand the dynamics of diffusion, as well as identify new research questions whose
empirical investigation can provide additional insights, especially into the potential barriers
of the diffusion of ADA.
The rest of the paper is organized as follows: Section 2 describes the objectives of the
framework to be developed and derives the basic design decisions. More specifically, we
apply the dimensions of socio-technical systems theory (technology, actors, rules) to the
diffusion of ADA. Since actors (auditors) reproduce rules and technologies in their activities,
we also draw upon the psychological theories to analyze the cognitive processes and
distortions/biases, as well as the affects. We explicitly consider the conceptual issues (e.g.,
the audit risk model and materiality), which are important for the re-shaping of the current
audit process. We also investigate the main drivers of the diffusion (e.g., audit quality).
Based on a preliminary analysis of 40 empirical ADA studies, Section 3 further opens the
black box of the categories (e.g., cognitive biases relevant in the use of ADA) developed in
Section 2. In this section, we refine the framework by developing and integrating additional
1
We prefer the general term ADA, rather than big data analytics, because our primary focus is on recent
developments (especially since 2010) in data analytics that may, or may not, use large data or entire data sets
and advanced techniques (e.g., predictive analytics, process mining).

sub-categories. In addition, we analyze the key findings of the previously mentioned studies
within and across the developed categories to derive future research perspectives. Section 4
provides the conclusions and discusses the limitations. By illustrating the insights that
emerge from our holistic perspective, the paper will be of interest to audit researchers,
standard setters and audit practitioners.
2 Methodology
2.1 Empirical research framework - Basic design decisions

The developed framework strives to fulfil the following objectives. Firstly, the framework
seeks to present the subject matter in a way that promotes an understanding through
categorization. It integrates the empirical findings and promotes more comprehensive
interpretations of these findings by considering additional perspectives. Building on this
foundation, the second objective is to direct future research, e.g., in cases of inconsistent or
contradictory evidence and when there are no or few findings for certain categories or a
combination of categories that should be investigated jointly. This course of action
stimulates new questions that, in turn, require the acquisition of new empirical knowledge,
as well as the modification of the framework itself. This is known as iterative heuristics. A
series of cycles characterizes this iterative process, and thus, functions as the interplay
between the conceptual (framework) and procedural (empirical research) components. Thus,
the framework presents a ‘medium of progress in science’ (Kubicek, 1977).
The diffusion of ADA takes place through the interplay among actors, technology, and
external rules. The auditors, as the primary actors, and the technologies they use are
entangled. The sociomateriality perspective (Fayard & Weeks, 2014) focuses on how the use
of ADA (technology regarded as a material object) by auditors (social refers to auditor’s
habits, beliefs, and perceptions) may re-configure the audit process. In sociomaterial terms,
the benefits of ADA come to the fore as auditors interpret the properties of ADA during
practice (Salijeni et al., 2021). Another process theory related attempt focuses on the steps
of the technology adoption process of ADA in audit firms and the contextual factors that
affect this process (Krieger et al., 2021). This perspective focuses on the identification of
use-cases in a very early stage of the diffusion of ADA without taking into account any re-
configurations of the audit process and the auditor’s judgment and decision making (JDM).
Overall, these perspectives are relatively vague.

Auditing depends on its ability to engender trust. For this reason, auditors are seeking
legitimacy, and in doing so, they are producing legitimacy (Power, 2003). Audit
technologies are often viewed as a quasi-scientific means to provide images of rationality.
Better than existing IT-audit tools, ADA is seen as a technical solution for the problem of
maintaining the legitimacy of the audit function (Salijeni, 2019; De Santis & D’Onza, 2021).
Legitimacy also plays an important role in the institutionalization process. Institutional
theory focuses on the institutional pressure organizations experience to incorporate ADA in
the audit process. Walker and Brown-Liburd (2019) consider this perspective and adapt an
innovation framework (Swanson & Ramiller, 2014) to describe how organizations handle
the emergence of ADA technologies in their day-to-day operations.
Each of the theoretical perspectives provides helpful insights. However, less light is shed
on the auditor’s JDM and the fact that audit practices are driven, to a large extent, by the
rules that enable or constrain ADA implementation. We use the socio-technical systems
theory (STT) presented in Geels (2004) to incorporate these perspectives. STT combines
institutional theory with sociology to describe the diffusion of technology. This theory
considers three interrelated dimensions: (1) technology, (2) rules, and (3) human actors and
organizations.
Technology does not work on its own. It works through the involvement of the actors and
their corresponding organizations. The actors apply the rules and reproduce them in their
activities. The technology shapes the rules and the rules may restrict or promote the use of
the technologies. The rules, therefore, play an important role in a highly regulated
environment, where each of these dimensions interacts. Hence, STT focuses on the dynamics
of the diffusion of the new technologies. Therefore, it is particularly useful to analyze the
shifts from one socio-technical system to another. “In reaction to new technologies, policy
makers may develop new rules to regulate it, and users may develop new behaviour” (Geels,
2004, 909). The consequence of these multiple games is that the dimensions co-evolve.
Therefore, we use these dimensions as a starting point for developing the first group of
categories for our framework (diffusion of ADA). 2
2
Austin et al. (2021) also uses the three STT categories, but without an in-depth investigation within these
categories and without further investigating the audit and cognitive processes.

In our paper, innovative technology refers to ADA. Current external rules 3 do not forbid,
but also do not explicitly require, the usage of ADA. Against this background, audit research
is important for the standard setter, especially when it comes to convincing the standard
setter that certain ADA-related changes are beneficial and assist in the implementation of
these changes in the standards. A successful innovation in a highly regulated domain also
requires a willingness and commitment of the standard setter (Curtis et al., 2016). Applying
ADA technologies also requires certain professional competencies, 4 which should become
subject to educational standards.
The actors and their organizations primarily refer to the audit firm and the audit client. 5
The audit firms are both the producers and users of the ADA technologies. Following the
technology acceptance model (Davis et al., 1989), the actual use of ADA and its acceptance
depends on the perceived usefulness and the ease of use. In particular, platforms used by the
audit client (e.g., Enterprise Resource Planning) enable the auditor to apply ADA tools more
efficiently (Eilifsen et al., 2020; Salijeni et al., 2019), which makes the use of ADA attractive
for meeting the commercial goals of the audit firms. Therefore, it is necessary to take a closer
look at the auditor and the audit team using these technologies, as well as the applied audit
approach, which embodies the organizational rules guiding the audit process.
It is important to focus the research on the dynamic interactions between these dimensions
to better understand their co-evolution. This will help guide the research in a field that is still
in a process of change. The benefits of ADA, as the main driver of its diffusion, do not exist
objectively in the technologies themselves. Rather, the benefits attributed to ADA are real-
ized through the actions taken by the auditors (actors) concurrent with technology adoption
(Fischer, 1996). Therefore, the auditors continuously reproduce rules and technologies while
3
This paper focuses on the audit of annual financial statements during the current institutional regime. We
primarily refer to the International Federation of Accountants (IFAC) and its boards, especially the
International Standards on Auditing (ISAs) published by the International Auditing and Assurance Standards
Board (IAASB), the International Code of Ethics published by the International Ethics Standards Board for
Accountants (IESBA), and the International Education Standards (IES) published by the International
Accounting Education Standards Board (IAESB). IFAC membership consists of 180 professional accountancy
organizations across 135 countries and jurisdictions. In general, members are directly responsible for adopting
and implementing these standards (https://www.ifac.org, accessed June 2, 2022).. In addition, activities by
national standard setters, especially in Canada (e.g., Chartered Professional Accountants in Canada, CPA
Canada), the United Kingdom (e.g., Financial Reporting Council, FRC) and the United States (e.g., the
American Institute of Certified Public Accountants, AICPA), that explore the growing use of data analytics in
the audit are taken into account.
4
Professional competence involves the integration and application of technical competence, professional skills,
and professional values, ethics, and attitudes. Please refer to IAESB, Framework for International Education
Standards for professional accountants and aspiring professional accountants.18.
5
To open the black box of the auditor (as an actor) in more detail, we focus on the auditor-related cognitive
processes in Section 3.2.2.

conducting the audit, which leads to our second group of categories: actors reproducing rules
and technologies.
The auditor's actions are based on the cognitive processes. As such, it is essential to
analyze how ADA influences the auditor's cognitive processes, and thus, the audit quality
and efficiency, because this, in turn, is likely to influence the diffusion of ADA. More
specifically, a deeper understanding of auditors’ judgment and decision making (JDM)
reveals the possible risks and benefits of the use of ADA. It also highlights the potential
regulatory and organizational needs. In other words, cognitive processes are regarded as a
means through which environmental, personal and task factors, affect JDM.
Consequently, our framework uses psychological theories to study JDM issues in ADA.
As a starting point, we consider the theoretical information processing approach (Newell &
Simon, 1972). Information processing relates to cognitive operations through which
information is identified, aggregated, evaluated, and used to draw conclusions. The task
environment represents the objective structure of the audit task as the problem to be solved.
Like other human beings, auditors are boundedly rational. In less structured or unstructured
tasks (e.g., ADA), the auditor can only perceive, select, and use a limited amount of
information to assess a particular matter. Accordingly, the auditor builds up a limited,
subjective problem space. The problem-solving activities (i.e., problem representation,
hypothesis generation, information gathering, conclusions) take place as a heuristic search
in this problem space. Thus, the auditor is prone to cognitive biases. For this reason, we
additionally follow the heuristics-and-biases-approach of psychology, which has been
widely adopted in the auditing research (Birnberg & Shields, 1989; Bonner, 2008; Tversky
& Kahneman, 1974) to further develop the categories of interest. Moreover, affects effect
the cognitive processing activities, with affect and cognitive biases interacting.
Radical innovations often take place in niches (Geels, 2004). Through the lens of a
framework, niches are specific sub-steps of the audit process (e.g., the use of ADA for fraud
detection). Due to the diffusion of ADA, the current audit process is likely to change, such
that the associated process steps, their design, and arrangement are also subject to change.
However, there is currently no clear outline for how the future audit process will be
structured.
To scientifically support the change, we introduce a supporting sub-category, which
focuses on the process-related conceptual issues (e.g., the audit risk model). These issues
have to be resolved to lay the groundwork for the further design and arrangement of the

process steps. This category has a temporary character in that its rationale for existence is
closely linked to overcoming the conceptual challenges.
The conditions of the audit market and the expectation gap, as well as the audit quality
and efficiency (i.e., commercial perspective of audit firms; see Power, 1999), are the primary
drivers of developments in audit regulations (e.g., audit quality issues in the Wirecard
scandal). As the audit profession claims to serve the public interest (IFAC, 2012), these
drivers can be regarded as conditions under which audits of financial statements can produce
the outcomes of public interest (e.g., low expectation gap, high audit quality). These drivers
also help us to better understand the dynamic relationships within the framework. Fig. 1
presents the framework.
[Insert Figure 1 here]
2.2 Search for empirical studies
The search for studies included the following two steps. Firstly, we covered all studies by
searching for keywords closely related to data analytics (advanced analytics, big data, data
analytics, information technology, data mining, process mining, visualization, social media)
and auditing (audit, and assurance). As a starting point, we considered all articles (final or
online early publications) in selected journals 6 and cross-checked the studies identified in
the papers’ reference sections. In addition, we conducted a comprehensive SSRN- (Social
Science Research Network), Scopus-, Google- and Google Scholar search and searched
articles in additional journals, books, and dissertations. We also searched IFAC, significant
regional and national professional organizations, and audit firms websites. Due to the nature
of the topic being in a dynamic process of change, we also considered publicly available
working papers as timely research contributions. If the research results were first published
as a dissertation, then subsequent publications in journals with very similar content were not
considered. In addition, unpublished material and studies not written in the English language
were not taken into account.
6
Accounting and Business Research, Accounting Horizons, Accounting, Auditing & Accountability Journal,
Accounting, Organizations and Society, Advances in Accounting Behavioral Research, Auditing: A Journal of
Practice & Theory, Behavioral Research in Accounting, Contemporary Accounting Research, European
Accounting Review, International Journal of Accounting, International Journal of Accounting Information
Systems, International Journal of Auditing, Journal of Accounting Research, Journal of Accounting, Auditing
and Finance, Journal of Emerging Technologies in Accounting, Journal of Information Systems, Managerial
Auditing Journal, Production and Operations Management, and The Accounting Review.

Secondly, we analyzed whether or not these studies are, in fact, closely related to data
analytics in auditing and contain a recognizable and comprehensible contribution to the
auditing knowledge. The empirical studies must fulfill the minimum quality standards for
qualitative or quantitative empirical research. This means that there should at least be a clear
description of the study design, the units of study, the means through which they were
selected, the data collection process, and the development of measurements and
classifications. A quantitative study should state what statistical analyses were conducted
and the appropriateness of the tests. Descriptive and inferential statistics should be provided
for each essential analysis. A qualitative study, in particular, should present the process of
obtaining evidence to justify any assertion or interpretation. 7 We acknowledge that the
elimination of studies due to substandard quality is, to a certain extent, subjective.
The first search process was conducted in September of 2019. We extended the search in
January of 2021. Cited working papers were reviewed in September of 2021 for a more
recent version or publication. Appendix A summarizes the 40 relevant empirical studies that
were identified in the search process.
3 Empirical research framework - Categories and analysis
Building on the basic design decisions in Section 2 and a preliminary analysis of the
empirical studies, the black boxes of the categories that have been developed are further
opened and the related findings of the current empirical ADA research are discussed. Future
research questions are assigned to a certain category due to their primary character, even if
their investigation addresses several categories. Appendix B summarizes the selected
research questions and rough outlines of the potential empirical research designs to address
these questions.
3.1 Diffusion of ADA
3.1.1 Innovative technology
Big data analytics is an innovative technology comprising new data sources and new
advanced techniques for data analyses. Audit clients have begun to make use of data in a
variety of ways, as the quantity of data increases and as structured data is more readily
7
Elimination is based on the standards for reporting on empirical research studies, as published by the
American Educational Research Association (2016).
10

available in real-time or at the day-to-day level. Unstructured, often non-financial data from
outside the client organization is also used more frequently. This can include textual data
(e.g., emails, web pages, Twitter, Facebook, and Google), as well as videos, images, and
audio data (Alles & Gray, 2016; Teoh, 2018). The use of this disaggregated client data
contributes significantly to the discovery of anomalies, which are more effective in detecting
anomalies at a daily level than on a weekly, monthly, or annual level. In this case, the use of
disaggregated data is a primary driver of audit effectiveness (Kogan et al., 2014).
Data analytics techniques comprise data visualization, process and text mining (Zhaokai
& Moffitt, 2019), clustering, regressions, simulations, machine learning, causal analytics,
predictive analytics, and social network analysis (Gepp et al., 2018). For example, process
mining of event logs analyzes audit-related processes and identifies control risks (Werner et
al., 2021). Identified exceptions from the expected process steps can be further analyzed
using Benford’s Law or social network analysis (Jans et al., 2014; Jans, 2019). Using
external data that cannot be easily manipulated by the client often produces effective
auditors’ expectations. For example, comments posted on Twitter can improve the prediction
of firm-level sales and the detection of errors in sales (Bartov et al., 2018; Tang, 2018). Deep
autoencoder neural networks help detect local anomalies in journal entries (Schreyer et al.,
2019).
There are several recent audit applications of data analytics developed by the big audit
firms for their own application and some general software applications. More specifically,
Deloitte uses a process X-ray as a process-mining tool, while PricewaterhouseCoopers
(PwC) uses the visualization tool Halo for journal entry testing and Ernst & Young (EY)
Helix is a library of data analytics. Overall, data analytics for journal entry testing is widely
used (Financial Reporting Council (FRC), 2017). General applications comprise
applications for different purposes and applications tailored to the audit. For example, there
is the general visualization tool, Tableau (Hoelscher & Mortimer, 2018) and Statistical
Analytics Software (SAS), a tool used for data mining and predictive analysis. General audit
applications include Celonis, Galvanize, Interactive Data Extraction and Analysis (IDEA),
WizRule, and AppZen. However, a comprehensive overview of the general audit tools is not
currently available.
As expected, empirical audit-related work is rare. This can be explained by the early stage
of development of ADA tools and the intention of audit firms to signal their strong
commitment in this area without providing too much proprietary information. Further
11

research work should focus on the development and practical use of ADA tools and the
embedded techniques. Particularly promising fields include the use of disaggregated client
data and the identification of further specific audit areas for the sequential or simultaneous
use of different techniques. Another promising research methodology to explore the
development, use, and understanding of data analytics in the audit domain is design science
research. 8
3.1.2 External rules
External rules in the field of auditing correspond to education standards, technical standards,
quality standards, and ethical standards. At present, the focus is primarily on educational and
technical standards and, to some extent, ethical standards. Thus, there is a research gap with
regard to quality standards.
Educational standards: New skills required in the use of advanced ADA are commonly
discussed (Sledgianowski et al., 2017; Appelbaum et al., 2021). Auditors often perceive that
inadequate technical and statistical knowledge impedes the use of data analytics (CPA
Canada, 2017), where visualization skills seem to be the most important (KPMG, 2018).
Based on a literature review, the International Accounting Education Standards Board
(IAESB) 9 made amendments to the International Education Standards (IES) 2, 3, 4, and 8
corresponding to changed or additional information and communications technologies skills
(IAESB, Staff, 2019). Such changes to the learning outcomes in professional development
are due, in part, to the need ‘to evaluate data and information from a variety of sources and
perspectives through research, integration, and analysis’ (IES 3.7(a)(i)). In addition, the
auditor has to demonstrate the ability to ‘respond effectively to changing circumstances or
new information to solve problems, inform judgments, make decisions, and reach well-
reasoned conclusions’ (IES 3.7(a)(v)), a skill that is referred to in the standard-setting
process as intellectual agility.
Further research is needed to determine a useful measure of intellectual agility, which is
similar, but not identical, to learning agility or cognitive flexibility. We need to know more
about this intellectual skill, particularly about whether or not such agility can be trained or
8
Design science research is an information technology research methodology (Gregor & Hevner, 2013).
From the perspective of design science research, new data sources, techniques of data analysis, as well as audit
evidence and assurance concepts can be regarded as artifacts that are useful for solving practical problems.
See, for example, Kogan et al., (2019) and Werner and Gehrke (2019), with regard to a new process mining
based approach for testing internal controls.
9
An International Panel on Accounting Education will replace the IAESB.
12

whether it must be considered as an unchangeable personality trait. Furthermore, in
acquiring IT competencies, certain personal traits (e.g., high tolerance for ambiguity) are
important (Lamberton et al., 2005). How should ADA competencies be considered in initial
(CPA exam) and continuing professional development? In this field, there is a need to
provide scientific support to these proposals and to develop a holistic educational
perspective.
Current technical standards do not constrain auditors in their use of ADA. However, there
are significant uncertainties surrounding how to use especially big data as audit evidence
(Appelbaum et al., 2017; Walker & Brown-Liburd, 2019). For example, the question of
whether or not it is possible to obtain appropriate and sufficient audit evidence (ISAs 330.18;
500.6) solely based on ADA needs to be investigated. This might be possible if all relevant
data from mass customer contracts are electronically available (e.g., energy suppliers). 10
Audit sampling (ISA 530) may become obsolete under these rare circumstances.
Appropriateness is the measure of the quality of the audit evidence (i.e., its relevance and
reliability; see ISA 500.7). This leads to another question: whether and under what
conditions can external data, in particular, satisfy the requirements of reliability? As stated
in the US Statement on Auditing Standard (SAS) 142.A27, reliability is affected by the
following attributes: accuracy, completeness, authenticity, and risk of bias. Uncertainties in
judging whether or not these attributes are valid lead to the fear that, in a setting where
conducting tests about details is mandatory, the sole use of ADA may not be permissible and
could cause sanctions (CPA Canada, 2019; Eilifsen et al., 2020). In addition, auditors often
consider the concept of sufficiency as significantly underdeveloped (Brown et al., 2020).
Similarly, there are uncertainties in the standards regarding the testing of internal controls
by means of ADA. One intention of the test of controls is to compensate for the risk of audit
sampling. If the substantive testing of a full population is possible by using ADA, the
question of whether or not the tests of controls are necessary is relevant (Institute of
Chartered Accountants in England and Wales (ICAEW), 2016, 19; IAASB, DAWG, 2017).
The current empirical research supports the notion that ADA is useful in obtaining the
evidence of the effectiveness of internal controls (Chiu, 2018; Jans et al., 2014). To avoid
ineffective over-auditing (Eilifsen et al., 2020; Salijeni, 2019), the standard setter ought to
10
The US auditing standard SAS 142.A69, which superseded AU-C 500 in July 2020, also considers it
permissible to waive the test of details under certain circumstances. US standards and ISA are often
comparable. A revision of ISA 500 is planned for 2023.
13

make a clear statement in this regard. Here, the conditions necessary for the use of ADA as
a test of internal control should be clarified (Werner & Gehrke, 2019).
Although the technical standards do not constrain the use of advanced ADA, the given
uncertainties do limit their use. If technical standards contained an obligation to use data
analysis (under certain circumstances), it would lead to a further diffusion and pave the way
for more innovative applications of data analytics.
At the current early stage of diffusion, it is difficult to make clear and reliable
recommendations for whether or not and how the technical standards need to be changed.
Further research is needed to provide evidence in the improvements in audit efficiency and
the quality that the standard setter requires for potential changes to technical standards. This
proof is difficult to produce, because the current state of the standards hampers the use of
advanced ADA, which leads to a kind of lock-in situation supporting the stability of the
current technical standards.
Another stabilizing factor is the alignment between individual technical standards (Geels,
2004). In the case at hand, it is difficult to change one auditing standard without altering the
other standards. When considering setting rules-based vs. principles-based standards, the
latter seems to be preferable in dynamic and complex environments. This raises the question
of whether or not more or less principles-based technical data analytics standards are
preferable (Austin et al., 2021).
Ethical standards: The auditor has the sole responsibility for the audit opinion expressed
(ISA 620.3). As such, the auditor must possess appropriate professional competence
(International Code of Ethics, Subsection 113). Therefore, the auditor has to evaluate the
adequacy of the ADA as audit evidence. Consequently, further research should focus upon
this aspect and the competencies needed. Auditors may rationalize unethical decisions by
displacing responsibility to ADA which discourages critical thinking (Westermann et al.,
2015). Thus, it is necessary to investigate whether or not data analysis options provided by
the system hinder the auditor to develop an appropriate audit trail and whether or not
professional skepticism is impaired.
3.1.3 Actors and their organizations
Actors and their organizations primarily refer to the audit firm and the audit client. ADA
leads to a stronger link between human and machine skills, from which follows a more
cooperative audit process. Such a process requires a team of auditors and other actors,
including specialists (e.g., data analysts), to work interactively to fulfill the audit task.
14

Competencies to decide on the use of ADA tools and their application are often distributed
among different actors. Therefore, team composition is an important research topic to
investigate (Omoteso, Patel, & Scott, 2010).
From an organizational perspective, the audit firm’s approach represents organizational
rules guiding the audit process. ADA, as part of the audit approach, might improve
organizational success and effectiveness, and thus, might increase the diffusion of ADA.
Referring to a macro-institutional perspective, ADA is interpreted as a way to externally
legitimize the organization’s existence (Tolbert & Zucker, 1996). From a micro-institutional
perspective (Zucker, 1987), ADA, as part of the audit approach, legitimizes auditors’
behavior within the audit firm.
The rules of the audit firm are especially important in the current environment, which has
relatively few imprecise external rules that do not explicitly prescribe the use of ADA. Audit
research has already examined the question of an appropriate structure for the assurance
approach and the right balance between structure and judgment (Broberg, 2013; Cushing &
Loebbecke, 1986). This question is also of particular interest in the context of the use of
ADA (e.g., several ADA tools are portrayed as providing structure to the audit approach; see
Salijeni, 2019, 143).
Closely related to legitimization considerations is the potential impact of ADA on the
expectation gap (e.g., Ruhnke & Schmidt, 2014). The enhanced expectations of the audit
client and related audit stakeholders could impose the use of ADA to auditors. However, the
use of ADA may be misleading with regard to the expectation that testing 100% of a
population provides something more than a reasonable assurance opinion (IAASB, DAWG,
2016). This might increase the expectation gap and could have serious consequences for
auditors’ reputation and litigation. Here, whether or not ADA leads to a widened expectation
gap and how this can be closed to better inform the public should be investigated.
The use of audit technology varies among audit firms of all sizes (Austin et al., 2021;
Eilifsen et al., 2020). From the perspective of resource dependency theory, smaller audit
firms only have limited capacities to develop ADA tools and must therefore apply general
tools. However, small audit firms tend to audit smaller clients, for which the prerequisites
for the use of data analytics are less often provided. This is a topic that should be analyzed
in more detail. A subsequent interesting topic for research is the question of whether or not
small firms’ limited access to ADA will increase market concentration and restrict
15

competition, or whether actors like SAP and Google are potential competitors in the audit of
clients with a high degree of digitization.
Moreover, ADA can also provide new business insights and can stimulate clients to
change their practices (Austin et al., 2021). If the auditor slips into a more advisory role, this
could result in threats of independence (International Code of Ethics, Section 600 ff.).
Notwithstanding this fear, auditors feel that ADA can help resolve disagreements between
the clients and the auditors (Kend & Nguyen, 2020). Hence, future interview based research
should investigate the impact of ADA on the auditor-client relationship, especially in relation
to the aforementioned points.
With regard to the competencies required for using (advanced) ADA, audit firms need to
train their current workforce. Small audit firms’ access to ADA training opportunities plays
a critical role in the auditor’s ADA use (Hampton & Stratopoulos, 2016). One general
challenge is to find the right set of competencies for a single auditor, as well as for an audit
team size and composition. The initial empirical evidence suggests that effective use depends
on at least one enthusiastic individual audit team member with partner support (FRC, 2017).
As illustrated previously, there are several starting points for future research on the actors
and organizations involved in the diffusion of ADA. Examples include the composition of
engagement teams (Dagiliene & Kloviene, 2019), the implementation of ADA in the audit
approach, auditor-client relationships and negotiations.
3.2 Actors reproducing rules and technologies
Overall, the diffusion of ADA reconfigures the audit procedures and changes the way audit
evidence is obtained. To support a potential re-organization of the current audit process and
its institutional environment, we add a sub-category focusing on the conceptual issues we
expect to be important for re-shaping the current audit process. Additionally, to better
understand and improve auditors’ JDM and to obtain insights into the potential barriers to
the diffusion of ADA, we consider the cognitive processes related to the use of ADA
separately.
3.2.1 Audit process
The audit process follows several steps, and because of its iterative nature, it is difficult to
identify a sequence of detailed process steps. As current international technical audit
standards follow a certain structure (IAASB, 2020), we follow this structure as a starting
16

point to identify the key steps in performing an audit. Continuous documentation
accompanies these steps. We discuss the use of ADA concerning the following four key
steps. 11
(1) Identifying and assessing the risks of material misstatements (RMM): ADA tools help
to identify and assess the RMM through a better understanding of the entity and its
environment, including the entity’s internal controls (ISA 315.19 ff.). Exploratory data
analysis techniques (e.g., data visualization, cluster analysis, data mining, text mining,
process mining, social network analysis) can be used for this purpose (Alawadhi, 2015;
Appelbaum et al., 2021; Liu, 2014). For example, data mining can contribute to the
understanding of the client (Gray & Debrecency, 2014), while weather data can help to
develop an expectation for retail revenues or to reduce sales forecast errors (Steinker et al.,
2017; Yoon, 2016), and employee postings on social media are useful in predicting growth
in key income statement information (Hales et al., 2018).
Process mining is an effective tool for analyzing business processes. However, when
examining the entire population of data, process mining can create problems of its own, due
to a huge number of exceptions from an expected process flow (Chiu, 2018; Jans et al.,
2014).
Very large databases always contain arbitrary correlations due to the size, not the nature,
of the data (Calude & Longo, 2017). Therefore, audit specific methods to reduce false
positives (so-called exceptional exceptions) have to be developed (Issa, 2013; No et al.,
2019). Future research should address this issue.
(2) Designing and performing procedures responsive to identified risks: According to the
ISA 330, the auditor has to perform procedures responsive to the identified risks. If all
relevant audit data is available electronically in an appropriate format, it seems possible to
obtain sufficient and appropriate audit evidence (ISA 500.6) by only using ADA. This open
issue is intended to be considered in the revision of the ISA 500, as discussed previously (in
the external rules Section). If the auditor’s assessment of an RMM at the assertion level
includes an expectation that the internal controls are operating effectively, the auditor shall
perform the tests of control (ISA 330.8). Whether the tests of controls are necessary or not
while testing the full population is not addressed in the current audit standards. Therefore,
11
Appelbaum et al., (2018, pp. 99-100) presented a framework for external audit analytics focusing on the
audit phases and analytic techniques. The intention is to organize existing research and to provide guidance for
new research. Based on a primarily technology-focused literature review, recommendations are made as to
which techniques should be further investigated at which phase.
17

we recommend that whether and under which conditions the tests of controls are dispensable
should be investigated experimentally.
(3) Procedures to address specific circumstances: ADA provides the opportunity to
derive reliable input for going-concern models (Austin et al., 2021; Gepp et al., 2018).
Another frequent application relates to the use of ADA to detect fraud (Austin et al.,
2021), where different approaches can be used (e.g., neural networks, process mining; see
Gepp et al., 2018; Gray & Debrecency, 2014). In addition, new datasets and data analysis
techniques offer new opportunities. Chiu et al., (2020) provide evidence that large
discrepancies between Google searches for a firm’s product and reported sales growth signal
a higher frequency for revenue manipulation. Auditors can use this signal for their risk
assessment and determination of further audit procedures responsive to the identified risk.
Perols et al. (2017) illustrate that the use of new data analytics preprocessing methods
improves fraud prediction. Byrnes (2019) offers a solution for automated clustering to
improve fraud detection, stating that inconsistent patterns between accounting performance
and electricity consumption is a useful indicator for detecting fraud (Allee et al., 2021).
Further research should investigate the usefulness of other external information (that cannot
be manipulated by the client) to detect fraud. Reducing false positives in fraud detection is
another interesting issue (Baader & Krcmar, 2018).
Large data provides better input for accounting estimates (e.g., allowances for doubtful
accounts, warranty obligations, goodwill, and intangible assets) (Austin et al., 2021).
Predictive analytics can help in assessing forward-looking assumptions. However, current
research in this area is rare.
Further examples include data generated by the use of drones, which can be used as
evidence for the existence and valuation of inventories and fixed assets (e.g., oil pipelines)
(Appelbaum & Nehmer, 2017); radio frequency identification technology that can provide
new data for inventory valuation (Krahel & Titera, 2015); and data generated by blockchain
technology that could enable automatic confirmations (Richins et al., 2017), which might
affect ISA 505. These examples should be investigated in more detail. Further potential areas
to address specific circumstances has to be identified and examined with an emphasis on the
possible effects on audit quality (Section 3.2.2).
(4) Audit conclusions and reporting: The ISA 700 does not address ADA. Therefore, it
may be beneficial to provide further guidance on how ADA can assist auditors to accumulate
audit evidence to express an overall opinion. Research related to cognitive processing is
18

expected to be helpful (e.g., to explore cognitive distortions). Another challenge is to identify
key audit matters that need to be communicated in the auditor’s report (ISA 701), such as if
the analysis of identified correlations requires significant attention by the auditor.
Current process-related research is strongly focused on the status quo. For this reason, we
consider conceptual issues that might be important for re-shaping the current audit process
separately. These issues relate to the concept of audit evidence, the audit risk model, the
assertion concept, the concept of assurance, and the concept of materiality.
Current technical standards require the auditor to obtain sufficient and appropriate audit
evidence. Research can contribute to a better understanding of how using ADA, especially
unstructured data, as audit evidence affects sufficiency, relevance, and reliability
considerations (Yoon et al., 2015). As shown in Kogan et al. (2014), disaggregated data
improves its predictive ability and the detection performance of anomalies. Disaggregated
data can improve the relevance of audit evidence. When data is continuously available, the
processes generating that data must be continuously assured (Krahel & Titera, 2015).
Furthermore, external data is often not as reliable as assumed. Garcia Lara et al. (2006)
illustrate significant differences in databases (e.g., Datastream, Worldscope). Research
should also help to answer the question of whether or not a high volume of data can
compensate for lower reliability (Appelbaum et al., 2017). Data from social media could be
biased. For example, Twitter users are not representative of the population of the customer
base (Yoon et al., 2015).
The standard setter assumes that the audit risk model (ARM), as currently defined in the
ISAs, is fundamentally sound (IAASB, DAWG, 2016; IAASB, Staff, 2018). The ARM that
guides the audit process is commonly defined as follows: Audit Risk (AR) = Risk of Material
Misstatement (RMM = Inherent Risk, IR × Control Risk, CR) × Detection Risk (DR). ADA
may significantly increase IR and CR (Appelbaum, 2016), and under certain conditions, the
sampling risk as part of the DR will be low or non-existent. However, when using the entire
population of transaction data, new risks (e.g., outlier risk) become evident. Therefore, the
character of the audit process might change from a sample-driven process to a more data-
driven one (Appelbaum et al., 2017).
IR and CR inversely relate to DR. The basic idea of the ARM is to determine IR and CR
first and then to conduct audit procedures to reduce DR to an acceptable level. The use of
ADA relates to risk identification and assessment (IR), substantive procedures including
analytical procedures and tests of details (DR), and tests of control (part of CR). Thus,
19

performing ADA procedures might be useful to determine IR and CR, on the one hand, and
on the other hand, to reduce DR simultaneously. Conceptual and empirical work should
address whether or not the ARM in its current form is still useful. For example, there is need
to investigate whether or not a multiplicative linkage between the components of the ARM
is suitable enough to provide a high quality risk assessment (Section 3.2.2).
Future research should explore the guiding ideas the auditor should follow while using
ADA. Besides the issue of whether or not the ADA can replace tests of details, another
interesting question is how to reduce the outlier risk to an acceptable low level (i.e., meeting
the requirement of ISA 520.5(c)). If the precision increases while using ADA, fewer tests of
details seem to be necessary.
The assertion concept (ISA 315.28b, 315.A190) also reflects the basic idea of the ARM.
One aim of this concept is that assertions typically represent risky areas, and, in this way, it
helps to focus the auditor's attention on them. Future research should address this issue: Is
this concept still useful in the context of large data or entire data sets? Do the special features
of the use of ADA lead to changes in the existing assertions? Is it required to better align the
design and use of ADA to the current assertions? In addition, there is preliminary evidence
that ADA is more effective for the completeness and accuracy assertion in comparison to
the other assertions (CPA Canada, 2017). These issues require a more detailed investigation.
The level of assurance is the level of certainty that the auditor has obtained after
conducting the audit. There are different concepts of assurance (e.g., limited assurance,
reasonable assurance). In conducting an annual audit of financial statements, the auditor has
to obtain reasonable assurance. In a reasonable assurance engagement, the auditor has to
reduce AR to an acceptably low level in the circumstances of the engagement (IFAC
Framework.14). The required levels of assurance in an individual audit engagement vary
according to the characteristics of the underlying subject matter. More specifically, whether
or not the information about a subject matter is historical or prospective, relates to a point in
time, or covers a period. Such characteristics affect the precision with which the underlying
subject matter is measured and the persuasiveness of the available evidence (IFAC
Framework.40). Thus, these characteristics determine the maximum level of assurance
(varying between 0 and 100%) in an individual case (Ruhnke & Lubitzsch, 2010). Here,
access to relevant data represents a characteristic of the subject matter that might increase
the required assurance level (in %), when compared to an audit without using this data. This
20

issue needs to be further explored. Future research should also address whether or not
another (new) assurance concept might fit better.
Data analytics presents a challenge in the current steps of the application of the concept
of materiality, as described in ISA 320. This involves investigating whether or not there is a
need to determine performance materiality (IAASB, DAWG, 2017, comment by EY). Future
research should examine the application of the concept of materiality when identifying
outliers, analyzing disaggregated data, or using data analytics tools (e.g., process mining).
Here, initial research attempts have focused on how to use materiality thresholds for the risk
prioritization of process instances (Chiu, 2018).
3.2.2 Cognitive processes
Following the theoretical arguments in Section 2.1, it is important to focus on the underlying
cognitive processes and related cognitive biases. Moreover, affects effect cognitive
processing activities. Furthermore, the environmental factors, task factors, and personal
factors determine auditor’s JDM. As shown in Fig. 1, all of these sub-categories and factors
affect the audit quality and efficiency, as they are important drivers of the ADA diffusion.
As a starting point, the auditor generates an initial representation of the problem based on
prior information of the client, as well as other knowledge from long-term memory. This
problem representation provides a road map for completing the task. When using large data,
the amount of immediately available initial data from the task environment increases
enormously. Thus, it might be more challenging to elicit effective representation in the
problem space. Consequently, several questions arise. These include which organizing
principles to consider (i.e., the most relevant cue, a pattern of cues, risks, assertions) for
problem representation in an unstructured and complex setting; what factors influence the
choice of problem representation; and should these principles vary according to specific
audit tasks?
Problem representation is important, because auditors often have difficulty shifting from
an unproductive initial problem representation to a productive one (Bierstaker et al., 1999).
Anderson et al. (2020) find that the categorization of identical visualization evidence as a
risk assessment procedure or substantive analytical procedure (ISAs 315, 520) impacts
auditor’s JDM. Individuals often derive internal representations from external frames.
Categorization, as set out in the audit standards, might impair audit quality. Therefore, the
impact of different forms of problem representation on the subsequent audit process and
audit quality needs to be examined.
21

The problem representation guides the auditor’s generation of hypotheses (e.g., a pattern
of information indicating that the valuation of a certain balance sheet item is not plausible)
and directs the further search for information. Hypotheses generation is crucial in JDM.
More specifically, auditors who do not adjust an initially wrong hypothesis, often fail to
search for the contradicting evidence, and fail to update their beliefs for the alternative
hypotheses. Intellectual agility is an important issue in this regard (see the external rules
Section). Hence, future research should address how the use of ADA might affect the
generation of hypotheses (e.g., how to identify relevant audit patterns in time series data; see
Brown-Liburd et al., 2015). Another open research question is the development of initial
hypotheses based on ambiguous information (Luippold & Kida, 2012).
A large amount of data affects the search for information (i.e., obtaining audit evidence).
In a traditional setting, the auditor uses prior information to generate hypotheses when
performing analytical procedures. The auditor then has to obtain audit-relevant data to
support or refute these hypotheses (Koonce, 1993). When using exploratory ADA, the steps
of ‘hypothesis generation’ and ‘information search,’ as part of the hypotheses testing, are
even more closely related, or even reversed in the chronological sequence. In an ADA
setting, the auditor often has to generate an audit-related explanation for an identified
correlation. Further research should examine these issues.
Information search and the subsequent generation of an audit-related explanation often
trigger the search for further information, especially for conducting substantive procedures.
In light of new evidence, there is need for belief-updating. For complex tasks, belief-
updating is a sequential process, as described in belief-adjustment models (Hogarth &
Einhorn, 1992). If the auditor has reached the required level of belief (reasonable assurance),
the search for information is terminated and the auditor’s conclusion is formulated.
New ADA-related research questions include: Are belief-adjustment models appropriate
in this context? Are these models appropriate when using a long series of information? In
particular, the use of exploratory ADA will lead to a more iterative audit process. Therefore,
research should focus on the duration and sequence of the information retrieval process and
the evaluation activities, especially in relation to the timing of the ADA. Research should
also consider which encoding mode is preferable: the additive mode, the averaging mode or
another mode? With regard to the termination of ADA, Rose et al. (2017) illustrates that it
is better to examine ADA visualizations after forming an initial hypothesis. This result
22

suggests that auditors may fail to identify relevant patterns unless they first form initial
expectations about an audit matter. This point should be investigated in more detail.
JDM quality and efficiency are important drivers of the diffusion of ADA and proxies of
these drivers are often considered in the empirical research. Audit quality is much debated
and there are different views about what constitutes audit quality (Knechel et al., 2013;
Simnett & Trotman, 2018). Factual quality refers to whether auditors detect material
misstatements in financial statements and appropriately act on the discovery (DeAngelo,
1981). The cognitive process, cognitive biases and affects, as well as the environmental,
personal, multi-person, and task factors are related to the audit quality. Following the
heuristic potential of our framework, each category (isolated, combined with other
categories) is a candidate to be analyzed in terms of the ADA-related audit quality issues.
For example, further research should investigate how to measure the effects of different
forms of visualization on audit quality, extending the work of Anderson et al. 2020, and Holt
and Loraas, 2021.
Another important issue is the perception of audit quality, which ADA has been shown
to improve (Eilifsen et al., 2020, Table 5). Barr-Pulliam et al. (2022) illustrate that jurors
assess auditors to be less negligent if they use advanced audit analytics tools in comparison
with traditional audit techniques (contrary with regard to predictive modeling; see Ballou et
al., 2021). In this study, the use of ADA is correlated with perceptions of higher audit quality.
These perceptions help legitimize the use of advanced ADA, and thus, accelerate its
diffusion. However, some ADA approaches used by auditors are perceived by the external
reviewers to be lower in quality (Emett et al., 2020). This contradictory evidence requires
further research. In addition, ADA might be associated with the expectation of 100% testing
(i.e., high-quality perceptions), and thus, give rise to the illusion that an auditor can provide
absolute assurance for the financial statement as a whole. Hence, the research should focus
on identifying a respective expectation gap and the possible measures to close it.
The rationale of efficiency is based on offering the required audit quality at the lowest
cost. Christ et al., 2021 exceptionally shows efficiency improvements in inventory audits
based on the additional use of data from drones (Section 3.2.1). Although ADA is often
perceived to improve audit efficiency (Alles & Gray, 2016; Eilifsen et al., 2020; Salijeni,
2019), further research in validating this perception is still needed. One question that could
be asked is how the use of ADA impacts the audit report lag (Abernathy et al., 2017).
23

Auditors often use heuristics as simple ‘rules of thumb’ to arrive at conclusions in highly
complex and ambiguous audit tasks. These are essential characteristics of the use of
(advanced) ADA. In these cases, auditors are prone to cognitive biases (Kleinman et al.,
2010). A valuable source of inspiration for identifying and investigating these anomalies is
the heuristics-and-biases approach, proposed by Tversky & Kahneman (1974). The
heuristics are useful because they facilitate the discovery of satisfactory solutions in complex
problem spaces. However, the use of heuristics can lead to severe and systematic errors.
As shown in Fig. 1, the potential triggers for cognitive biases are the audit firm, the audit
client, the features of the audit task, the ADA tool/technique, and the cognitive processes.
The Association of Chartered Certified Accountants (ACCA, 2017) state that data analytics
could reduce cognitive biases. Upon a closer look, some biases may be mitigated, other
biases may change in nature, and new biases may appear. Potential biases include (Brown-
Liburd et al., 2015; Chang & Luo, 2019; Huerta & Jensen, 2017):
(1) Availability biases: The availability of information is determined by the ease of its
retrieval. Auditors might overestimate the importance of information that is easily available
(such as information stored in electronic databases).
(2) Confirmation and dilution biases: Due to their size, very large databases always
contain arbitrary correlations that (apparently) support management assertions, that are
embodied in the financial statements. If the auditor uses these correlations erroneously or
selectively as audit evidence to confirm management’s assertions, the auditor’s judgment
may be impaired. The vast amount of potentially irrelevant data often leads to a dilution
effect (Hackenbrack, 1992), given that adding irrelevant or weak evidence to existing strong
positive evidence reduces the overall strength of a belief about a hypothesis.
(3) Recency bias: Auditors might overemphasize the importance of more recent
information and neglect historical information.
(4) Inattentional blindness: Auditors might be prone to focus on the results of ADA.
However, not all audit-relevant information leaves its traces in the electronic data. Therefore,
auditors might be blind to other signals or overemphasize the audit evidence obtained from
the use of ADA (e.g. in the fraud context; see Kleinman and Anandarajan, 2011).
(5) Source credibility neglect: Research with regard to neglecting source credibility is
mixed and rare (Bamber, 1983; Joyce & Biddle, 1981). Technical audit standards require
considering the credibility (reliability) of the source, which is of particular importance for
the use of high volume and high variety data as audit evidence.
24

Debiasing aims to mitigate or overcome judgment shortcomings. Debiasing should be
attached to the cause of the judgment distortion. These strategies could be: data-related (e.g.,
debiasing by directing attention to relevant data and redirecting attention away from
irrelevant data); technique-related (e.g., debiasing by considering the biases triggered, such
as by a certain visualization technique); effort-related (e.g., debiasing by motivating subjects,
such as imposed penalties); or cognition-related (e.g., debiasing by training auditors to use
another mindset in certain settings) (Arkes, 1991). Debiasing could be further achieved by
modifying the standards and training activities, implementing documentation requirements,
and using judgment frameworks or decision aids. Debiasing research related to ADA use is
in its infancy. The following examples indicate areas for future debiasing research.
It is generally possible to find correlations in high volume data supporting different initial
assumptions by the auditor. This is often referred to as the auditor’s initial judgment
hypothesis. This may result in a corroborative information search, which could harm audit
quality. The IAASB has recognized this distortion and has stated in the recently revised ISA
315.13 that ‘[t]he auditor shall design and perform risk assessment procedures in a manner
that is not biased towards obtaining audit evidence that may be corroborate or towards
excluding audit evidence that may be contradictory’. Hence, further research should
investigate whether or not the proposed standard update is an effective debiasing mechanism.
Another debiasing mechanism is the use of a suitable cognitive mindset to debias the
judgment shortcomings (Austin et al., 2021). Future research should address this issue, while
also considering that research on cognitive mindsets in relation to ADA should control for
the technical competencies of the participants. To mitigate the dilution effect of irrelevant
information in the context of fraud risk assessments, it is useful to present information as
natural frequencies instead of probabilities (Eilifsen et al., 2019). In addition, whether or not
consistent rewards are effective in debiasing for non-skeptical actions triggered by higher
rates of false positives should also be an area of investigation (Barr-Pulliam et al., 2020).
Affects (comprising emotions and moods) have the potential to inappropriately influence
auditors’ JDM. Affects can also trigger a search for information, and thus, fulfill a
motivational function. In this respect, affects can also have positive effects. Affects can be
triggered by the actors of the audit firm, the audit client, the features of the audit task or the
use of ADA. Affects can influence risk perceptions and estimations, and audit opinions
(Chung et al., 2008; Cianci & Bierstaker, 2009).
25

Cognitive distortions and affects interact. Affects could also be a strong driver of
heuristics (Griffith et al., 2016). The dynamics between cognitive distortions, affects, and
heuristics is an open research question. Further research should also focus on the
competencies in handling the negative effects resulting from the affects (so called affect
regulation; see Daff et al., 2012).
3.2.3 Environmental, personal, multi-person and task factors
Environmental factors surround individuals while they conduct audits, such as incentives,
accountability, feedback, teams, pressure, information technology, external rules (Bonner,
2008; Libby & Luft, 1993). Environmental factors can have positive or negative influences
on auditors’ judgment. For example, a moderate level of time pressure increases audit
quality, a high level induces dysfunctional behavior (Pietsch & Messier, 2017).
Auditors reviewing graphical versus textual evidence are more skeptical of aggressive
management assumptions underlying complex estimates (Backof et al., 2018). Thus, data
visualization tools can help to debiase the client pressure. In addition, different graphical
displays can create visual illusions that lead to biased decisions (Amer & Ravindran, 2010).
Arnott (2006) illustrates that system developers are prone to personal preferences and
cognitive biases. Thus, system biases can be built into the system by design. For example,
system designers’ decisions, especially in the development of automated algorithms to
transform unstructured raw data into structured data for quantitative analysis, can bias
auditors’ JDM while using the system (Huerta & Jensen, 2017).
The diffusion of ADA has also led to the increasing automation of audit procedures and
auditor’s judgments (Salijeni, 2019). There is a risk that technology might dominate the audit
process, especially in the case of ADA usage with embedded artificial intelligence
components (Koreff et al., 2021). In this scenario, users (auditors) do not develop the
expertise that normally comes with experience, and eventually, all future auditors could lack
experience. These users, in turn, fail to learn from the system, an effect often referred to as
the deskilling effect of system usage. From a cognitive perspective, the development of
expertise by the user and the system is a challenge. To counteract the effects of technology
dominance and over-reliance, future tools should promote the development of the user’s
expertise. To that end, tools should be designed to better leverage the collaboration of the
user and the technology, where both contribute to the JDM process (Sutton et al., 2018).
In the field of knowledge-based audit systems, Arnold et al., (2018) show that the use of
a system interface that promotes expert knowledge representation has a significant positive
26

effect on the development of expert-like knowledge structures. Similar results can be found
in Rose et al., (2012). Based on interviews and experimental research it should be
investigated if there are features of ADA systems that can promote the development of the
user’s expertise.
Further research can also focus on the factors affecting the reliance on ADA. This
research should aim to identify the features that determine the auditor’s reliance, such as the
degree of interaction and the auditor’s cognitive style. Other relevant factors include the
impact of the use of instructions (e.g., the instruction to counter-explain) or the effects of
receiving feedback on the quality of the ADA-supported JDM (Bonner, 2008). The examples
provided are not exhaustive. However, the environmental factors mentioned previously,
together with our analysis in relation to the audit process and the cognitive processes, shed
light on questions that require closer examination.
Personal factors relate to the characteristics the individual auditor brings to the audit task
and the cognitive processes. Personal characteristics influence the auditor’s behavior and
distinguish one individual auditor from others. There is a variety of different views and
concepts of personality in the psychological literature. From a cognitive viewpoint, the
following factors seem to be important with regard to the ADA use in audit practice.
(1) Abilities: General problem-solving ability includes the ability to recognize
relationships, interpret data, and reason analytically (Libby & Luft, 1993). To some extent,
ability is likely to be innate and partially refined through experience (Bonner & Lewis,
1990). Ability often correlates positively with a high audit quality (Kallunki et al., 2019;
Kleinman et al., 2010). Audit firms and the standard setter often refer to the similar term
competencies. 12 Gao et al. (2021) find that data analytic competencies improve audit quality.
We see potential to investigate what kind of competencies (e.g., intellectual agility) are
relevant for using ADA. These competencies should be considered in the educational
standards.
(2) Intrinsic motivation relates to motives that drive auditors to action. Rose et al. (2021)
illustrate that visualizations that increase arousal enhance the auditors’ ability to recognize
the disconfirming audit evidence. This, in turn, correlates with an increase in the auditor’s
revenue adjustments. Overall, the motivation is only rarely researched in the context of the
usage of ADA.
12
See Footnote 4. Different approaches and terms (e.g., abilities, competencies, and intelligence) are used in
Sternberg and Kaufman (1998).
27

(3) Knowledge structures refer to how auditors organize information in the long-term
memory (see the cognitive processes Section). For example, the previous research has shown
that auditors tend to structure their knowledge of financial statement misstatements with the
audit objective as the primary organizing dimension (Coyne et al., 2010). Potential research
questions include: Which organizing dimensions are effective in the ADA context, for
example, when using process-mining techniques in a scenario with a vast number of
exceptions? How should auditors organize their knowledge along certain classes of
deviations (Hosseinpour & Jans, 2019)?
(4) Cognitive styles describe a person’s typical form of perceiving, thinking, and problem-
solving. Cognitive styles are relatively stable over time. Audit-related research focuses, for
example, on the relationship between cognitive style (analytic vs. intuitive) and task
performance (Fuller & Kaplan, 2004). Locus of control comprises the degree to which people
believe that their actions directly influence the outcomes (internal vs. external). Auditors
who have an internal locus of control perform better in audit firms using an unstructured
audit approach (Hyatt & Prawitt, 2001). Further studies should investigate which kind of
cognitive style or locus of control is preferable for using ADA, for example, when analyzing
unstructured data. Another issue relates to the preferable cognitive style or control concept
when using the work of ADA experts or artificial intelligence-based ADA systems
(Commerford et al., 2021).
(5) Tolerance for ambiguity refers to an individual’s predisposition and attitude toward
ambiguity, which affects information processing. Pincus (1990) illustrates that ambiguity-
intolerant auditors were significantly more likely to make a ‘not fairly presented’ decision,
which is consistent with the actual case. The unstructured nature of certain data, its variety
of formats, lack of source reliability, and imprecise causal knowledge creates ambiguity. In
this environment, only auditors with a high tolerance for ambiguity might be able to perform
effectively (Brown-Liburd et al., 2015; Vasarhelyi et al., 2015). ADA-related research needs
to address these issues: For example, besides the impact on audit quality, it should be
investigated whether this tolerance is an unchangeable personal trait or can be learned by
training (see Section 3.1.2).
(6) Metacognition refers to cognition about cognition. Plumlee et al. (2015) show that
two metacognitive skills (divergent and convergent thinking) are beneficial for performing
creative analytical procedures. These skills lead to a greater likelihood of generating and
choosing the correct explanation, which is an important issue in the ADA context. In
28

addition, metacognition skills are trainable, and thus, research in this field is interesting from
an educational perspective.
Multi-person factors are also a concern, as audits are often conducted by audit teams
whose members work interactively and concurrently. Due to the increasing importance of
specialist knowledge of data analytics and the iterative nature of the audit process, multi-
person processes are a promising field for further cognitive research (although not referring
to ADA, see Bamber et al., 1996; Trotman et al., 2015). Currently, there is no research
available on ADA in the context of audit team composition and team decision making (i.e.,
diffusion of ADA).
One particularly promising avenue for research in this realm is audit firms’ internal
quality control. In this process, big data provides more evidence that is both supportive and
contradictive, to evaluate the JDM of the audit engagement under review. This, in turn,
makes the review process more challenging. Further research is needed and might impact
the audit approach and external quality standards.
Task factors: Human information processing is highly contingent upon task
characteristics (Bonner, 2008). Different audit tasks require different abilities, knowledge,
and effort for their successful completion (Libby & Luft, 1993). Task factors relate to task
complexity and task structure. Beyond a particular level, task complexity generally has a
negative effect on audit quality (Chung & Monroe, 2001). ADA increases task complexity
by producing a larger number of information cues. Thus, it is cognitively challenging to
determine effective information-gathering activities and to come to a justifiable task output.
Future ADA-related research should be directed to address the complexity of the task
structure and its effects on evidence integration and belief revision (Kerr & Ward, 1994).
For example, benefits from visualizations using graphs or tables may depend on the
complexity of the task (Hwang & Wu, 1990; Vessey, 1991). Additionally, further research
can investigate whether or not knowledge is an important moderator of high task complexity
(Tan et al., 2002). Another issue is how audit quality in complex and dynamic tasks relates
to auditor mindsets, intrinsic motivation, and less structured audit approaches (Griffith et al.,
2015; Kadous & Zhou, 2019; Stuart & Prawitt, 2012).
4 Conclusions
ADA holds the potential to change audit practices fundamentally, despite its rare use in audit
practice currently. The literature often identifies ADA as a promising stream for future
29

research, but it is challenging to conduct this research, as the field is undergoing such
fundamental and dynamic changes.
The developed framework and the chosen categories provide information on the current
state of the empirical research and is intended to stimulate investigations into new research
questions. This will, in turn, broaden our preliminary understanding and the further
development of the framework. Thus, the framework represents a ‘medium of progress in
science’ (Kubicek, 1977). It illustrates how research and the accompanying advances in
knowledge occur through the interplay between a conceptual (framework) and a procedural
(empirical research) component. The multidimensional holistic framework we have
developed covers categories related to the diffusion of ADA (the technology, the actors using
it, and the external rules that regulate it) and the actors reproducing the rules and technologies
(focusing on the audit process, conceptual issues, and cognitive processes). In general, this
systematization is also transferable to other audit innovations.
Furthermore, the conceptual features of the framework were designed for its application
in a changing world. Significant drivers of fundamental changes in auditing are audit quality,
efficiency, the expectation gap and the conditions of the audit market. As this also applies in
the context of ADA, these drivers are included as framework categories. Another feature is
to consider separately conceptual issues related to the audit process that need to be resolved
to reshape the current process. Without a solution to these issues (e.g., those related to the
audit risk model and the concept of audit assertions), no in-depth changes in the audit
standards are to be expected. A further prerequisite is an in-depth understanding of the audit-
related cognitive processes, possible cognitive biases and the debiasing mechanisms in the
ADA context.
The framework enables empirical researchers to better comprehend this complex topic
and acquire an overview of the current state of the knowledge. In this way, the framework
contributes to an efficient research process. Due to the low diffusion rate of advanced ADA,
further exploratory interviews followed by experiments seem promising. In addition, case
studies and a design science approach are encouraged, for example, to investigate how single
tools or techniques can contribute to the detection of misstatements and to better understand
how to use these techniques to solve audit problems. In addition, this paper is of interest to
standard setters, in relation to the due process of setting international auditing standards,
which includes the requirement to develop positions based on appropriate research (so called
evidence-based policy making; see Salterio et al., 2021). The paper is also beneficial for
30

audit practitioners and various stakeholders of audited financial statements, as it helps to
better understand the current diffusion of ADA and the potential barriers of its use.
Based on our analyses, Appendix B presents selected research questions and rough
outlines of potential empirical research designs to address these questions. Although the
research questions can be assigned to a certain category, due to their primary characteristics,
the investigation usually addresses several categories. If, for example, an ADA tool is to be
empirically investigated with regard to its use by auditors, an obvious question arises as to
whether the risks, anomalies or misstatements are detectable by this tool. Thus, this
exploration addresses the category of innovative technology. Examples of further cross-
category related questions stimulated by the framework include: What competencies are
required of the auditor and other persons from the audit team to use the tool? How does the
tool interact with the traditional features of an audit? How should the tool be integrated into
the audit approach of the audit firm? How is the evidence obtained positioned in the risk
model and how can it contribute to the required level of assurance? How should materiality
considerations be considered? How does the use of the tool impact auditors’ JDM, especially
problem representation, hypothesis generation, and information retrieval? Do ADA-specific
cognitive biases (e.g., availability bias, confirmation bias, inattentional blindness, source
credibility neglect) exist? How should debiasing mechanisms be designed? Here, the
framework acts as a medium to systematically support the identification of these questions.
Another more specific research question derived from our analysis is whether or not the
use of ADA can serve simultaneously as a risk assessment procedure, as a substantive
procedure, and as a test of controls. Since the definition of the types of audit procedures
already influence the problem representation, the question arises as to whether or not the
existing types of procedures, which are closely connected to the components of the current
risk model, are appropriate in the ADA context. Should the audit risk model be redesigned,
particularly with reference to the possibility that the audit process might become more
iterative? Should the chronological sequence of the steps of the ‘hypothesis generation’ and
‘information search’ be reversed? For example, what is the best timing of the consideration
of ADA procedures in the audit process, before or after the generation of the hypotheses?
Which competencies (e.g., intellectual agility) are required and how can they be trained?
Our paper is not without limitations. Firstly, the elimination of studies due to minor
quality is, to a certain extent, subjective. On the other hand, the general exclusion of non-
reviewed contributions (e.g., working papers) would probably have impaired the
31

development of the framework and its heuristic potential. Secondly, another point of
criticism may be the lack of analytical depth within each category. We have deliberately
accepted this limitation, because a broad treatment of the topic and, in particular, the
promotion of cross-category considerations, has been given priority over a more detailed
examination of the individual categories. Hence, future conceptual work should focus on
analyzing separate categories or groups of closely related categories.
Despite these limitations, our paper illustrates that bringing different perspectives
together is important to improve our understanding of data analytics in the audit domain and
to identify innovative patterns of interpretations. A multitude of new research questions
based on the framework have been derived. However, the question of whether or not the
framework is suitable cannot be conclusively proven a priori. Future research results and
potential corresponding improvements in understanding will show whether the developed
framework is fruitful or not.
Declarations of interest: none
References
Abernathy, J. L., Barnes, M., Stefaniak, C., & Weisbarth, A. (2017). An international perspective on
audit report lag: a synthesis of the literature and opportunities for future research. International
Journal of Auditing, 21, 100-127. https://doi.org/10.1111/ijau.12083.
ACCA (2017). Banishing bias? Audit, objectivity and the value of professional scepticism. London.
Retrieved from https://www.accaglobal.com/vn/en/professional-insights/global-
profession/banishing-bias.html. Accessed June 2, 2022.
Alawadhi, A. (2015). The application of data visualization in auditing. Dissertation, Newark, New
Jersey. Retrieved from https://rucore.libraries.rutgers.edu/rutgers-lib/47630/. Accessed June 2,
2022.
Allee, K. D., Baik, B., & Roh, Y. (2021). Detecting financial misreporting with real production
activity. Contemporary Accounting Research, 38 (3), 1581-1615. https://doi.org/10.1111/1911-
3846.12674
Alles, M. G., & Gray, G. L. (2016). Incorporating big data in audits: identifying inhibitors and a
research agenda to address those inhibitors. International Journal of Accounting Information
Systems, 22, 44-59. https://doi.org/10.2308/acch-51067.
Amer, T. S., & Ravindran, S. (2010). The effect of visual illusions on the graphical display of
information. Journal of Information Systems, 24 (1), 23-42.
https://doi.org/10.2308/jis.2010.24.1.23.
32

American Educational Research Association 2006. Standards for reporting on empirical social
science research in AERA publications. Educational Researcher, 35 (6), 33-40. Retrieved from
https://journals.sagepub.com/doi/10.3102/0013189X035006033. Accessed June 2, 2022.
Anderson, S. B., Hobson, J. L., & Peecher, M. E. (2020). The joint effects of rich data visualization
and audit procedure categorization on auditor judgement. Working Paper. Retrieved from
https://scholarspace.manoa.hawaii.edu/items/6924754c-07bd-45fe-8fc5-f3cafbc209a3. Accessed
June 2, 2022.
Appelbaum, D. (2016). Securing big data provenance for auditors: the big data provenance black box
as reliable evidence. Journal of Emerging Technologies in Accounting, 13 (1), 17-36.
.https://doi.org/10.2308/jeta-51473.
Appelbaum, D., Kogan, A., & Vasarhelyi, M. A. (2017). Big data and analytics in the modern audit
engagement: research needs. Auditing: A Journal of Practice & Theory, 36 (4), 1-27.
https://doi.org/10.2308/ajpt-51684.
Appelbaum, D. A., Kogan, A., & Vasarhelyi, M. A. (2018). Analytical procedures in external
auditing: a comprehensive literature survey and framework for external audit analytics. Journal
of Accounting Literature, 40, 83-101. https://doi.org/10.1016/j.acclit.2018.01.001.
Appelbaum, D., & Nehmer, R. A. (2017). Using drones in internal and external audits: an exploratory
framework. Journal of Emerging Technologies in Accounting, 14 (1), 99-113.
https://doi.org/10.2308/jeta-51704.
Appelbaum, D., Showalter, D. S., Sun, T., & Vasarhelyi, M. A. (2021). A framework for auditor data
literacy: a normative position. Accounting Horizons, 35 (2), 5-25.
https://doi.org/10.2308/horizons-19-127.
Arkes, H. R. (1991). Costs and benefits of judgment errors: implications for debiasing. Psychological
Bulletin, 110 (3), 486-498. http://dx.doi.org/10.1037/0033-2909.110.3.486.
Arnott, D. (2006). Cognitive biases and decision support systems development: a design science
approach. Information System Journal, 16, 55-78. https://doi.org/10.1111/j.1365-
2575.2006.00208.x.
Arnold, V., Collier, P., Leech, S. A., Rose, J. M., & Sutton, S. G. (2018). Can knowledge-based
systems be designed to counteract deskilling effects? Working paper.
Austin, A., Carpenter, T., Christ, M., & Nielson, C. (2021). The data analytics journey: Interactions
among auditors, managers, regulation, and technology. Contemporary Accounting Research, 38
(3), 1888-1924. https://doig.org/10.1111/1911-3846.12680.
Baader, G., & Krcmar, H. (2018). Reducing false positives in fraud detection: combing the red flag
approach with process mining. International Journal of Accounting Information Systems, 31, 1-
16. https://doi.org/10.1016/j.accinf.2018.03.004.
Backof, A. G., Carpenter, T. D., & Thayer, J. (2018). Auditing complex estimates: how do construal
level and evidence formatting impact auditors’ consideration of inconsistent evidence.
Contemporary Accounting Research, 35 (4), 1798–1815. https://doi.org/10.1111/1911-
3846.12368.
Ballou, B., Grenier, J.H., & Reffett, A. (2021). Stakeholder perceptions of data and analytics based
auditing techniques. Accounting Horizons, 35 (3), 47-68, DOI:
https://doi.org/10.2308/HORIZONS-19-116.
Bamber, E. M. (1983). Expert judgment in the audit team: a source reliability approach. Journal of
Accounting Research, 21 (2), 396-412. https://doi.org/10.2307/2490781.
Bamber, E. M., Watson, R. T., & Hill, M. C. (1996). The effects of group support system technology
on audit group decision-making. Auditing: A Journal of Practice & Theory, 15 (1), 122-134.
33

Barr-Pulliam, D., Brazel, J. F., McCallen, J. B., & Walker, K. (2020). Data analytics and skeptical
actions: The countervailing effects of false positives and consistent rewards for skepticism.
Working Paper. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3537180.
Accessed June 2, 2022.
Barr-Pulliam, D., Brown-Liburd, H. L., & Sanderson, K. A. (2022). The effects of the internal control
opinion and use of audit data analytics on perceptions of audit quality, assurance, and auditor
negligence. Auditing: A Journal of Practice & Theory, 41 (1), 25-48,
https://doi.org/10.2308/AJPT-19-064.
Bartov, E., Faurel, L., & Mohanram, P S. (2018). Can Twitter help predict firm-level earnings and
stock returns? The Accounting Review, 93(3), 25-57. https://doi.org/10.2308/accr-51865.
Bierstaker, J. L., Bedard, J. C., & Biggs, S. F. (1999). The role of problem representation shifts in
auditor decision processes in analytical procedures. Auditing: A Journal of Practice & Theory, 18
(1), 18-36. https://doi.org/10.2308/aud.1999.18.1.18.
Birnberg, J. G., & J. F. Shields, J.F. (1989). Three decades of behavioral accounting research: A
search for order. Behavioral Research in Accounting (1), 23-74.
Brazel, J.F./Jones, K.L., & Lian, Q. (2022). Auditor use of benchmarks to assess fraud risk: The case
for industry data, Working Paper. Retrieved from
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3591263. Accessed June 2, 2022.
Bonner, S. E. (2008). Judgment and decision making in accounting. Upper Saddle River, New Jersey:
Pearson, Prentice Hall.
Bonner, S. E., & Lewis, B. L. (1990). Determinants of auditor expertise. Journal of Accounting
Research, 28 (Supplement), 1-20. https://doi.org/10.2307/2491243.
Broberg, P. (2013). The auditor at work, a study of auditor practice in Big 4 audit firms. Dissertation,
Lund. Retrieved from https://www.lu.se/lup/publication/606f4fd2-033a-4389-8b0b-
9d7b14db3d2e. Accessed June 2, 2022.
Brown, V. L., Dickins, D., Hermanson, D. R., Higgs, J. L., Jenkins, J. G., Nolder, C., Schaefer, T. J.,
& Smith, K. W. (2020). Comments of the Auditing Standards Committee of the Auditing Section
of the American Accounting Association on proposed Statement on Auditing Standards (SAS),
Audit evidence. Current Issues in Auditing, 14 (1), C1-C9, https://doi.org/10.2308/ciia-52603.
Brown-Liburd, H., Issa, H., & Lombardi, D. (2015). Behavioral implications of big data’s impact on
audit judgment and decision making and future research directions. Accounting Horizons, 29 (2),
451-468. https://doi.org/10.2308/acch-51023.
Byrnes, P. E. (2019). Automated clustering for data analytics. Journal of Emerging Technologies in
Accounting, 17 (2), 43-58. https://doi.org/10.2308/jeta-52474.
Byrnes, P. E., Criste, T. R., Stewart, T. R., & Vasarhelyi, M. (2015). Reimagining auditing in a wired
world. In AICPA (ed.), Audit analytics and continuous audit (87-103). New York: AICPA.
Calude, C. S., & Longo, G. (2017). The deluge of spurious correlations in big data. Foundations of
Science, 22 (3), 595-612. https://doi.org/10.1007/s10699-016-9489-4.
Cao, M., Chychyla, R., & Stewart, T. (2015). Big data analytics in financial statement audits.
Accounting Horizons, 29 (2), 423-429. https://doi.org//10.2308/acch-51068.
Chang, C. J., & Luo, Y. (2019). Data visualization and cognitive biases in audits. Managerial
Auditing Journal, https://doi.org/10.1108/MAJ-08-2017-1637.
Chiu, T. (2018). Exploring new audit evidence: the application of process mining in auditing.
Dissertation, Newark, New Jersey. Retrieved from https://rucore.libraries.rutgers.edu/rutgers-
lib/57303/PDF/1/play/. Accessed June 2, 2022.
34

Chiu, P. C., Teoh, S. H., Zhang, Y., & Huang, X. (2020). Using Google searches of firm products to
assess revenue quality and detect revenue management. Working paper. Retrieved from
Christ, M. H./Emett, S. A./Summers, S. L., & Wood, D. A. (2021). Prepare for takeoff: improving
asset measurement and audit quality with drone-enabled inventory audit procedures. Review of
Accounting Studies, 26, 1323-1343, https://doi.org/10.1007/s11142-020-09574-5.
Chung, J., & Monroe, G. S. (2001). A research note on the effects of gender and task complexity on
an audit judgment. Behavioral Research in Accounting, 13 (1), 111-125.
https://doi.org/10.2308/bria.2001.13.1.111.
Chung, J. O. Y., Cohen, J. R., & Monroe, G. S. (2008). The effect of moods on auditors’ inventory
valuation decisions. Auditing: A Journal of Practice & Theory, 27 (2), 137-159.
https://doi.org//10.2308/aud.2008.27.2.137.
Cianci, A. M., & Bierstaker, J. L. (2009). The impact of positive and negative mood on the hypothesis
generation and ethical judgments of auditors. Auditing: A Journal of Practice & Theory, 28 (2),
119-144. https://doi.org/10.2308/aud.2009.28.2.119.
Commerford, B., Eilifsen, A., Hatfield, R. C., Holmstrom, K., & Kinserdal, F. (2021). Control issues:
How providing input affects auditors’ reliance on artificial intelligence. Working paper.
Coyne, M. P., Biggs, S. F., & Rich, J. S. (2010). Priming/reaction-time evidence of the structure of
auditors’ knowledge of financial statement errors. Auditing: A Journal of Practice & Theory, 29
(1), 99-123. https://doi.org/10.2308/aud.2010.29.1.99.
CPA Canada (2017). Audit data analytics alert, survey on the use of audit data analytics in Canada.
Toronto. Retrieved from https://www.cpacanada.ca/en/business-and-accounting-resources/audit-
and-assurance/canadian-auditing-standards-cas/publications/audit-data-analytics-alert-ada-
survey-results. Accessed June 2, 2022.
CPA Canada (2019). An inside look at how auditors in Canada are using data analytics. Toronto.
Retrieved from https://www.cpacanada.ca/en/business-and-accounting-resources/audit-and-
assurance/canadian-auditing-standards-cas/publications/how-auditors-are-using-data-analytics.
Curtis, E., Humphrey, C., & Turley, W. S. (2016). Standards of innovation in auditing. Auditing: A
Journal of Practice & Theory, 35 (3), 75-98. https://doi.org/10.2308/ajpt-51462.
Cushing, B. E., & Loebbecke, J. K. (1986). Comparison of audit methodologies of large accounting
firms. Sarasota, Florida: American Accounting Organization.
Daff, L., de Lange, P., & Jackling, B. (2012). A comparison of generic skills and emotional
intelligence in accounting education. Issues in Accounting Education, 27 (3), 627-645.
https://doi.org/10.2308/iace-50145.
Dagiliene, L., & Kloviene, L. (2019). Motivation to use big data and big data analytics in external
auditing. Managerial Auditing Journal, 34 (7), 750-782. https://doi.org/10.11008/MAJ-01-2018-
1773.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A
comparison of two theoretical models. Management Science, 35 (8), 982-1003. https://
https://doi.org/10.1287/mnsc.35.8.982.
DeAngelo, L. E. (1981). Auditor independence, ‘low balling’, and disclosure regulation. Journal of
Accounting and Economics, 3 (2), 113-127. https://doi.org/10.1016/0165-4101(81)90009-4.
De Santis, F./D’Onza, G. (2021). Big data and data analytics in auditing: in search of legitimacy.
Meditari Accountancy Research, 29 (4), 1088-1112. https://doi.org/10.1108/MEDAR-03-2020-
0838.
35

Eilifsen, A., Kinserdal, F., Messier, W. F., & McKee, T. (2020). An exploratory study into the use
of audit data analytics on audit engagements, Accounting Horizons 34 (1), 75-103.
https://doi.org/10.2308/HORIZONS-19-121.
Eilifsen, A., Kochetova, N., & Messier, W. F. (2019). Mitigating the dilution effect in auditors’
judgments using a frequency response mode, Behavioral Research in Accounting, 31 (2), 51-71.
https://doi.org/10.2308/bria-52467.
Emett, S. A., Kaplan, S. E., Mauldin, E. G., & Pirckerd, J. S. (2020). Auditing with data and analytics:
External reviewer perception of audit quality and effort. Working paper. Retrieved from
Fayard, A. L., & Weeks, J. (2014). Affordances for practice. Information and Organization, 24 (4),
236-249. https://doi.org/10.1016/j.infoandorg.2014.10.001.
Fischer, M. J. (1996). ‘Real-izing’ the benefits of new technologies as a source of audit evidence: an
interpretive field study. Accounting, Organizations and Society, 21 (2/3), 219-242.
https://doi.org/10.1016/0361-3682(95)00005-4.
FRC (2017). Audit quality thematic review, The use of data analytics in the audit of financial
statements. London: FRC. Retrieved from https://www.frc.org.uk/getattachment/4fd19a18-1beb-
4959-8737-ae2dca80af67/AQTR_Audit-Data-Analytics-Jan-2017.pdf. Accessed June 2, 2022.
Fuller, L. R., & Kaplan, S. E. (2004). A note about the effect of auditor cognitive style on task
performance. Behavioral Research in Accounting, 16 (1), 131-143.
https://doi.org/10.2308/bria.2004.16.1.131.
Gao, T./Huang, S., & Wang, R. (2021). Data analytics and audit quality, Working paper. Retrieved
from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3928355. Accessed June 2, 2022.
Garcia Lara, J. M., Garcia Osma, B., & Noguer, B. G. de A. (2006). Effects of database choice on
international accounting research. ABACUS, 42 (3/4), 426-454. https://doi.org/10.1111/j.1467-
6281.2006.00209.x.
Gartner (2021). IT Gartner glossary. Retrieved from https://www.gartner.com/it-glossary/. Accessed
June 2, 2022.
Geels, F. W. (2004). From sectoral systems of innovation to socio-technical systems: Insights about
dynamics and change from sociology and institutional theory. Research Policy, 33 (6/7), 897-
920. https://doi.org/10.1016/j.respol.2004.01.015.
Gepp, A., Linnenluecke, M. K., O`Neill, T., & Smith, T. (2018). Big data techniques in auditing
research and practice: current trends and future opportunities. Journal of Accounting Literature,
40, 102-115. https://doi.org/10.1016/j.acclit.2017.05.003.
Gray, G. L., & Debreceny, R. S. (2014). A taxonomy to guide research on the application of data
mining to fraud detection in financial statement audits. International Journal of Accounting
Information Systems, 15 (4), 357-380. https://doi.org/10.1016/j.accinf.2014.05.006.
Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum
impact. MIS Quarterly, 37 (2), 337-355. http://dx.doi.org/10.25300/MISQ/2013/37.2.01 .
Griffith, E. E., Hammersley, J. S., Kadous, K., & Young, D. (2015). Auditor mindsets and audits of
complex estimates. Journal of Accounting Research, 53 (1), 49-77, https://doi.org/10.1111/1475-
679X.12066.
Griffith, E. E., Kadous, K., & Young, D. (2016). How insights from the ‘New’ JDM Research can
improve auditor judgment: fundamental research questions and methodological advice. Auditing:
A Journal of Practice & Theory, 35 (2), 1-22. https://doi.org/10.2308/ajpt-51347.
Hackenbrack, K. (1992). Implications of seemingly irrelevant evidence in audit judgment. Journal
of Accounting Research, 30 (1), 126-136. https://doi.org/10.2307/2491095.
36

Hales, J., Moon, J. R., & Swenson, L. A. (2018). A new era of voluntary disclosure? Empirical
evidence on how employee postings on social media relate to future corporate disclosures.
Accounting, Organizations & Society, 68-69, 88-108. https://doi.org/10.1016/j.aos.2018.04.004.
Hampton, C., & Stratopoulos, T. C. (2016). Audit data analytics use: an exploratory analysis.
Working paper. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2877358.
Hoelscher, J., & Mortimer, A. (2018). Using Tableau to visualize data and drive decision-making.
Journal of Accounting Education, 44, 49-59. https://doi.org/10.1016/j.jaccedu.2018.05.002.
Holt, T., & Loraas, T. M. (2021). A potential unintended consequence of “Big Data”: Does
information structure lead to suboptimal auditor judgment and decision-making? Accounting
Horizons, 35(3), 161-186. https://doi.org/10.2308/HORIZONS-19-123.
Hogarth, R. M., & Einhorn, H. J. (1992). Order effects in belief updating: the belief-adjustment
model. Cognitive Psychology, 24 (1), 1-55. http://dx.doi.org/10.1016/0010-0285(92)90002-J.
Hosseinpour, M., & Jans, M. J. (2019). Process deviation categories in an auditing context. Working
paper. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3280339. Accessed
June 2, 2022.
Huerta, E., & Jensen, S. (2017). An accounting information systems perspective on data analytics
and big data. Journal of Information Systems, 31 (3), 101-114. https://doi.org/10.2308/isys-
51799.
Hwang, M., & Wu, B. (1990). The effectiveness of computer graphics for decision support: a meta-
analytical integration of research findings. Database, 21 (2/3), 11-20.
https://doi.org/10.1145/109022.109023.
Hyatt, T. A., & Prawitt, D. F. (2001). Does congruence between audit structure and auditors’ locus
of control affect job performance? The Accounting Review, 76 (2), 263-274.
https://doi.org/10.2308/accr.2001.76.2.263.
IAASB (2020). Handbook of international quality control, auditing, review, other assurance, and
related services pronouncements, 2020 edition, Volume 1, New York. Retrieved from:
https://www.ifac.org/system/files/publications/files/IAASB-2020-Handbook-Volume-1.pdf.
IAASB, DAWG (2016). Exploring the growing use of technology in the audit, with a focus on data
analytics, request for input. New York. Retrieved from: https://www.ifac.org/publications-
resources/exploring-growing-use-technology-audit-focus-data-analytics. Accessed June 2, 2022.
IAASB, DAWG (2017). Exploring the growing use of technology in the audit, with a focus on data
analytics, submitted comment letters. New York. Retrieved from:
https://www.ifac.org/publications-resources/exploring-growing-use-technology-audit-focus-
data-analytics. Accessed June 2, 2022.
IAASB, Staff (2018). Exploring the growing use of technology in the audit, with a focus on data
analytics, Feedback Statement. New York. Retrieved from: https://www.ifac.org/publications-
resources/feedback-statement-exploring-growing-use-technology-audit-focus-data. Accessed
June 2, 2022.
IAESB, Staff (2019). International Education Standards, Basis for conclusions, Revisions to IESs 2,
3, 4, and 8 – Information and communications technologies and professional skepticism, New
York. Retrieved from: https://www.iaesb.org/publications/revisions-ies-2-3-4-and-8-1. Accessed
June 2, 2022.
ICAEW (2016). Data analytics for external auditors. London.
37

IFAC (2012). IFAC Policy Position 5, June 2012, A definition of the public interest, New York.
Retrieved from: https://www.ifac.org/system/files/publications/files/PPP%205%20(2).pdf.
Issa, H. (2013). Exceptional exceptions. Dissertation, Newark, New Jersey. Retrieved from:
https://rucore.libraries.rutgers.edu/rutgers-lib/41484/. Accessed June 2, 2022.
Jans, M. (2019). Auditor choices during event log building for process mining. Journal of Emerging
Technologies in Accounting, 16 (2), 59-67. https://doi.org/10.2308/jeta-52496.
Jans, M., Alles, M. G., & Vasarhelyi, M. A. (2014). A field study on the use of process mining of
event logs as an analytical procedure in auditing. The Accounting Review, 89 (5), 1751-1773.
https://doi.org/10.2308/accr-50807.
Joyce, E. J., & Biddle, G. C. (1981). Are auditor's judgments sufficiently regressive? Journal of
Accounting Research, 19 (2), 323-349. http://dx.doi.org/10.2307/2490868.
Kadous, K., & Zhou, Y. (2019). How does intrinsic motivation improve auditor judgment in complex
audit tasks? Contemporary Accounting Research, 36 (1), 108-131. https://doi.org/10.1111/1911-
3846.12431.
Kallunki, J., Kallunki, J. P., Niemi, L., & Nilsson, H. (2019). IQ and audit quality: do smarter auditors
deliver better audits? Contemporary Accounting Research, 36 (3), 1373-1416.
https://doi.org/10.1111/1911-3846.12485.
Kend, M., & Nguyen, L. A. (2020). Big data analytics and other emerging technologies: the impact
on the Australian audit and assurance profession. Australian Accounting Review, 30, 269-282.
https://doi.org/10.1111/auar.12305.
Kerr, D. S., & Ward, D. D. (1994). The effects of audit task and evidence integration and belief
revision. Behavioral Research in Accounting, 6, 21-42.
Kleinman, G., Anandarajan, A., Medinets, A., & Palmon, D. (2010). A theoretical model of cognitive
factors that affect auditors’ performance and perceived independence. International Journal of
Behavioral Accounting and Finance, 1 (3), 239-267. Retrieved from:
https://www.inderscienceonline.com/doi/abs/10.1504/IJBAF.2010.031318. Accessed June 2,
2022.
Kleinman, G., & Anandarajan, A. (2011). Inattentional blindness and its relevance to teaching
forensic accounting and auditing. Journal of Accounting Education, 29 (1), 37-49.
https://doi.org/10.1016/j.jaccedu.2011.08.002.
Knechel, W. R., Krishnan, G. V., Pevzner, M., Shefchik, L. B., & Velury, U. K. (2013). Audit quality:
insights from the academic literature. Auditing: A Journal of Practice & Theory, 32 (Supplement
1), 385-421. https://doi.org/10.2308/ajpt-50350.
Kogan, A., Alles, M. G., Vasarhelyi, M. A., & Wu, J. (2014). Design and evaluation of a continuous
data level auditing system. Auditing: A Journal of Practice & Theory, 33 (4), 221-245.
Kogan, A., Mayhew, B. W., & Vasarhelyi, M. A. (2019). Audit data analytics research – an
application of design science methodology. Accounting Horizons, 33 (3), 69-73.
https://doi.org/10.2308/acch-52459.
Köhler, A., & Quick, R. (2018). Editorial: opportunities for innovative auditing research.
International Journal of Auditing, 22 (3), 329-331. https://doi.org/10.1111/ijau.12141.
Koonce, L. (1993). A cognitive characterization of audit analytical review. Auditing: A Journal of
Practice & Theory, 12 (Supplement), 57-76.
KPMG (2018). Next generation audit, n/a. Retrieved from:
https://assets.kpmg/content/dam/kpmg/us/pdf/2018/10/kpmg-audit-forbes-nga-report.pdf.
38

Koreff, J., Weisner, M. & Sutton, S. G. (2021). Data analytics (ab)use in healthcare fraud audits.
International Journal of Accounting Information Systems, 42,
https://doi.org/10.1016/j.accinf.2021.100523.
Krahel, J. P., & Titera, W. R. (2015). Consequences of big data and formalization on accounting and
auditing standards. Accounting Horizons, 29 (2), 409-422. https://doi.org/10.2308/acch-51065.
Krieger, F./Drews, P./Velte, P. (2021). Explaining the (non-)adoption of advanced data analytics in
auditing: A process theory. International Journal of Accounting Information Systems, 41,
Kubicek, H. (1977). Heuristic frameworks and heuristic research designs as elements of a
construction strategy of empirical research (Heuristischer Bezugsrahmen und heuristisch
angelegte Forschungsdesigns als Elemente einer Konstruktionsstrategie empirischer Forschung).
In R. Köhler (Ed.), Empirical and action theoretical research concepts in business economics
(Empirische und handlungstheoretische Forschungskonzeptionen in der Betriebswirtschaftslehre)
(3-36), Stuttgart: C. E. Poeschel.
Lamberton, B., Fedorowicz, J., & Roohani, S. J. (2005). Tolerance for ambiguity and IT competency
among accountants. Journal of Information Systems, 19 (1), 75-95.
https://doi.org/10.2308/jis.2005.19.1.75.
Libby, R., & Luft, J. (1993). Determinants of judgement performance in accounting settings: ability,
knowledge, motivation, and environment. Accounting, Organizations and Society, 18 (5), 425-
450. https://doi.org/10.1016/0361-3682(93)90040-D.
Liu, Q. (2014). The application of exploratory data analysis in auditing. Dissertation, Newark, New
Jersey. Retrieved from: https://rucore.libraries.rutgers.edu/rutgers-lib/45656/PDF/1/play/.
Luippold, B. L., & Kida, T. E. (2012). The impact of initial information ambiguity on the accuracy
of analytical review judgments. Auditing: A Journal of Practice & Theory, 31 (2), 113-129.
Newell, A., & Simon, H. A. (1972). Human problem solving, Englewood Cliffs, New York: Prentice-
Hall.
No, W. G., Lee, K., Huang, F., & Li, Q. (2019). Multidimensional audit data selection (MADS): a
framework for using data analytics in the audit data selection process. Accounting Horizons, 33
(3), 127-140. https://doi.org/10.2308/acch-52453.
Omoteso, K., Patel, A., & Scott, P. (2010). Information and communications technology and
auditing: current implications and future directions. International Journal of Auditing, 14 (2),
147-162. https://doi.org/10.1111/j.1099-1123.2009.00410.x.
Perols, J. L., Bowen, R. M., Zimmermann, C., & Samba, B. (2017). Finding needles in a haystack:
using data analytics to improve fraud prediction. The Accounting Review, 92 (2), 221-245.
Pietsch, C. P. R., & Messier, W. R. (2017). The effects of time pressure on belief revision in
accounting: a review of relevant literature within a pressure-arousal-effort-performance
framework. Behavioral Research in Accounting, 29 (2), 51-71. https://doi.org/10.2308/bria-
51756.
Pincus, K. V. (1990). Auditor individual differences and fairness of presentation judgments.
Auditing: A Journal of Practice & Theory, 9 (3), 150-166.
Plumlee, R. D., Rixom, B. A., & Rosman, A. J. (2015). Training auditors to perform analytical
procedures using metacognitive skills. The Accounting Review, 90 (1), 351-369.
39

Power, M. (1999). The audit society: rituals of verification, 2nd ed., Oxford: Oxford University
Press.
Power, M. (2003). Auditing and the production of legitimacy. Accounting, Organizations and
Society, 28, 379-394. https://doi.org/10.1016/S0361-3682(01)00047-2.
Richins, G., Stapleton, A., Stratopoulos, T. C., & Wong, C. (2017). Big data analytics: opportunity
or threat for the accounting profession. Journal of Information Systems, 31 (3), 63-79.
https://doi.org/10.2308/isys-51805.
Rose, J. M., McKay, B. A., Norman, C. S., & Rose, A. M. (2012). Designing decision aids to promote
the development of expertise. Journal of Information Systems, 26 (1), 7-34.
Rose, A. M., Rose, J. M., Rotaru, K., Sanderson, K. A., & Thibodeau, J. C. (2021). Effects of data
visualization choices on psychophysiological responses, judgment and audit quality, Online early,
https://doi.org/10.2308/ISYS-2020-046.
Rose, A. M., Rose, J. M., Sanderson, K. A., & Thibodeau, J. C. (2017). When should audit firms
introduce analyses of big data into the audit process? Journal of Information Systems, 31 (3), 81-
99. https://doi.org/10.2308/isys-51837.
Ruhnke, K., & Lubitzsch, K. (2010). Determinants of the maximum level of assurance for various
assurance services. International Journal of Auditing, 14 (3), 233-255.
https://doi.org/10.1111/j.1099-1123.2009.00414.x.
Ruhnke, K., & Schmidt, M. (2014). The audit expectation gap: existence, causes, and the impact of
changes. Accounting and Business Research, 44 (5), 572-601.
https://doi.org/10.1080/00014788.2014.929519.
Salijeni, G. M. (2019). Big data analytics and the social relevance of auditing: an exploratory study.
Dissertation, Alliance Manchester Business School. Retrieved from:
https://www.research.manchester.ac.uk/portal/files/95492166/FULL_TEXT.PDF. Accessed
June 2, 2022.
Salijeni, G., Samsonova-Taddei, A., & Turley, S. (2019). Big data and changes in audit technology:
contemplating a research agenda. Accounting and Business Research, 49 (1), 95-119.
https://doi.org/10.1080/00014788.2018.1459458.
Salijeni, G., Samsonova-Taddei, A., & Turley, S. (2021). Understanding how big data technologies
reconfigure the nature and organization of financial statement audits: a sociomaterial analysis.
European Accounting Review, 30, 531-555, https://doi.org/10.1080/09638180.2021.1882320.
Salterio, S. E., Hoang, K., & Luo, Y. (2021). Communication is a two-way street: analyzing
approaches undertaken to systematically transfer audit research knowledge to policymakers.
Accounting, Organizations & Society, 94, 1-21. https://doi.org/10.1016/j.aos.2021.101265.
Schreyer, M., Sattarov, T., Schulze, C., Reimer, B. & Borth, D. (2019). Detection of accounting
anomalies in the latent space using adversarial autoencoder neural networks. Working paper.
Retrieved from: https://arxiv.org/pdf/1908.00734.pdf. Accessed June 2, 2022.
Simnett, R., & Trotman, K. T. (2018). Twenty-five-year overview of experimental audit research:
trends and links to audit quality. Behavioral Research in Accounting, 30 (2), 55-76.
https://doi.org/10.2308/bria-52138.
Sledgianowski, D., Gomaa, M., & Tan, C. (2017). Toward integration of Big Data, technology and
information systems competencies into the accounting curriculum. Journal of Accounting
Education, 38, 81-93. https://dx.doi.org/10.1016/j.jaccedu.2016.12.008.
Steinker, S., Hoberg, K., & Thonemann, U. W. (2017). The value of weather information for e-
commerce operations. Production and Operations Management, 26 (10), 1854-1874.
https://doi.org/10.1111/poms.12721.
40

Sternberg, R. J., & Kaufman, J. C. (1998). Human abilities. Annual Review of Psychology, 49, 479-
502. https://doi.org/10.1146/annurev.psych.49.1.479.
Stuart, I. C., & Prawitt, D. F. (2012). Firm-level formalization and auditor performance on complex
tasks. Behavioral Research in Accounting, 24 (2), 193-210. https://doi.org/10.2308/bria-50113.
Sutton, S. G., Arnold, V., & Holt, M. (2018). How much automation is too much? Keeping the human
relevant in knowledge work. Journal of Emerging Technologies in Accounting, 15 (2), 15-25.
https://doi.org/10.2308/jeta-52311.
Swanson, E. B., & Ramiller, N. C. (2014). Innovating mindfully with information technology. MIS
Quarterly, 28 (4), 553-583.
Tan, H. T., Ng, T. B. P., & Mak, B. W. Y. (2002). The effects of task complexity on auditors’
performance: the impact of accountability and knowledge. Auditing: A Journal of Practice &
Theory, 21 (2), 81-95. https://doi.org/10.2308/aud.2002.21.2.81.
Tang, V. W. (2018). Wisdom of crowds: cross-sectional variation in the informativeness of third
party-generated product information on Twitter. Journal of Accounting Research, 56 (3), 989-
1034. https://doi.org/10.1111/1475-679X.12183.
Teoh, S. H. (2018). The promise and challenges of new datasets for accounting research. Accounting,
Organizations and Society, 68-69, 109-117. https://doi.org/10.1016/j.aos.2018.03.008.
Tolbert, P. S., & Zucker, L. G. (1996). The institutionalization of institutional theory. In S. Clegg, C.
Hardy, & Nord, W. (Eds.), Handbook of organization studies (175-190), Vol. 1, Thousand Oaks,
CA: Sage.
Trotman, K. T., Bauer, T. D., & Humphreys, K. A. (2015). Group judgment and decision making in
auditing: past and future research. Accounting, Organizations and Society, 47, 56-72.
https://doi.org/10.1016/j.aos.2015.09.004.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science,
185 (4157), 1124-1131. http://dx.doi.org/10.1126/science.185.4157.1124.
Vasarhelyi, M. A., Kogan, A., & Tuttle, B. M. (2015). Big Data in accounting: an overview.
Accounting Horizons, 29 (2), 381-396. https://doi.org/10.2308/acch-51071.
Vessey, I. (1991). Cognitive fit: A theory-based analysis of the graphs versus tables literature.
Decision Sciences, 22 (2), 219-240. https://doi.org/10.1111/j.1540-5915.1991.tb00344.x.
Walker, K. R., & Brown-Liburd, H. (2019). The emergence of data analytics in auditing: Perspective
from internal and external auditors through the lens of institutional theory. Working paper.
Werner, M., & Gehrke, N. (2019). Identifying the absence of effective internal controls—an
alternative approach for internal control audits. Journal of Information Systems, 33 (2), 205-222.
Werner, M., Wiese, M., & Maas, A. (2021). Embedding process mining into financial statement
audits. International Journal of Accounting Information Systems, 41.
Westermann, K. D., Bedard, J. C., & Earley, C. E. (2015). Learning the “craft” of auditing: A
dynamic view of auditors’ on-the-job learning. Contemporary Accounting Research, 32 (3), 864-
896, https://doi:10.1111/1911-3846.12107.
Yoon, K. (2016). Three essays on unorthodox audit evidence. Dissertation, Newark, New Jersey.
Retrieved from: https://rucore.libraries.rutgers.edu/rutgers-lib/50579/. Accessed June 2, 2022.
Yoon, K., Hoogduin, L., & Zhang, L. (2015). Big data as complementary audit evidence. Accounting
Horizons, 29 (2), 431-438. https://doi.org/10.2308/acch-51076.
Zhaokai, Y., & Moffitt, K. C. (2019). Contract analytics in auditing. Accounting Horizons, 33 (3),
111-126. https://doi.org/10.2308/acch-52457.
41

Zucker, L. G. (1987). Institutional theories of organization. Annual Review of Sociology, 13, 443-
464. https://doi.org/10.1146/annurev.so.13.080187.002303.
42

Diffusion of ADA
Innovative technology
Actors and their organizations External rules
ADA
Audit firm Educational standards
- Audit team
- Audit approach
Technical standards
Audit client Ethical standards
Other actors and related Quality, liability, and

stakeholders other standards
Environmental, personal and multi-person factors
Actors reproducing rules and technologies

Audit process
Identifying Procedures Procedures Auditor’s

and assessing responsive to the to address specific conclusions
RMM identified risks circumstances and reporting
Continuous documentation
Re-shaping
Process-related conceptual issues
Cognitive processes
Task factors Task environment
Problem space (individual auditor)

Effects
Problem representation
Generation of hypothesis
Search for information
Auditor’s conclusion
Trigger
Cognitive biases Affects Not audit-
related
triggers
Debiasing Affect
Fig. 1. Audit data analytics - empirical research framework
43

Appendix A. Empirical studies relating to ADA
Study Data collection Focus and main findings

Research method
Data analysis
Allee et al. (2021) 6,257 firm-year observations from 2006- Growth in electricity consumption is an
2014, Kisvalue (comparable to indicator of revenue growth. Differences in
Compustat), Korea Electric Power revenue growth and electricity growth are
Corporation, Korea useful indicators for detecting financial
Archival research misreporting (proxied by accounting
restatements, qualified audit opinions, and
Regressions
regulatory enforcement actions).
Anderson et al. (2020) 214 auditors, US Auditors are more likely to identify client data
Experiment, 2x2x2, between subjects outliers as high risk when data is visualized in
a histogram format relative to a tabular
Descriptive analysis, logistic regressions
format, but only when prior audit procedures
are categorized as risk assessment procedures
rather than as substantive analytical
procedures.
Austin et al. (2021) 55 auditors, practitioners (managers), and Interviews support the idea of regarding
standard-setters, US technology (ADA), audit clients/auditors, and
Semi-structured interviews standard-setters as interrelated dimensions of
the diffusion of ADA. Auditors and audit
Qualitative analysis
clients see themselves as partners on a journey
together exploring new ADA tools.
Ballou et al. (2021) 201 jurors, 232 non-professional No difference across three audit procedures
investors, US (statistical sampling, complete population
Experiment, between subjects, 2x2x3 testing, and predictive modeling) in perceived
auditor competence of jurors and investor
Descriptive analysis, ANOVA (Analysis
willingness to invest in an audited company.
of Variance)
Nature of auditor’s procedures affect jurors’
auditor negligence verdicts (marginally higher
negligence in the predictive modeling
condition).
Barr-Pulliam et al. 113 audit seniors, US When false positive rates are higher, auditors’
(2020) Experiment, 2x2, between subjects skepticism levels are low. Consistent rewards
for skepticism (positive internal performance
Descriptive analysis, logistic regressions,
evaluation by their supervisors) improve the
planned comparisons, path analysis
skepticism of auditors. No positive effects of
rewards can be observed when the positive
rates are extremely high and additional
investigation will lead to costly skepticism.
Barr-Pulliam et al. 578 jury-eligible participants, US Jurors perceive higher audit quality and assess
(2022) Experiment, 2x2, between subjects auditors as less negligent if auditors employ
ADA tools compared to traditional auditing
t-test, mediation analysis
sampling techniques. This study focuses on
the internal controls over financial reporting
opinion.
44

Bartov et al. (2018) 3,604 firms, 869,733 tweets from 2009- Aggregated opinions from Twitter help predict
2012, GNIP, Compustat, US a firm’s earnings.
Archival research
Descriptive analysis, regressions
Brazel et al. (2022) 465 fraud and 101,071 control The results show that auditors rely heavily on
observations 1994-2014, AAER prior-year balances and relations within the
(Accounting and Auditing Enforcement client’s financial data as benchmarks when
Releases issued by the Securitiy developing expectations during planning. In
Exchange Commission), Compustat, contrast, the archival study shows that
EDGAR (Electronic Data Gathering, differences between a company’s revenue
Analysis, and Retrieval, database), US / growth and the revenue growth of its industry
30 audit professionals, US are the best fraud indicator.
Archival research / Survey
Regressions / qualitative analysis
Chiu (2018) One European Bank Process mining is useful for the evaluation of
Case study the effectiveness of internal controls.
Variant analysis, segregation of duty

analysis, personnel analysis, and
timestamp analysis
Chiu et al. (2020) 29,905 observations from 2004-2014, Google searches for a firm’s product (measured
Google search data, Compustat, US by the Google Trends search volume index as
Archival research a proxy for the implied demand for a firm’s
product) provide information about firm sales.
Large deviations between reported sales
growth and change in Google searches can
detect low revenue quality (revenue
manipulation).
Christ et al. (2021) Counting inventory (sheep and cattle) Drone-enabled inventory audit procedures
using drones vs. manual techniques improve efficiency and effectiveness of
Case study, design science, interviews inventory counting and quality of
documentation
Descriptive analysis, qualitative analysis
CPA Canada (2017) 394 auditors, Canada Survey on the use of data analytics, especially
Survey regarding nature and extent of use. Also
considers factors driving (e.g., client
Descriptive analysis
expectations) or impeding (e.g., technical and
statistical knowledge) the use of data
analytics.
CPA Canada (2019) 9 audit firms (referring to 11 audit Exploration into the nature of data analytics:
engagements), Canada the procedures used, matters affecting the use,
Interviews benefits and challenges of use, and potential
for use in future audits.
Dagiliene & Kloviene 21 audit firms, regulators, and audit Exploration of motivating factors (company-
(2019) clients, Lithuania related and institutional) for the use of ADA,
Interviews and the circumstances in which to use ADA.
45

Eilifsen et al. (2019) 113 auditors, 58 students, Norway Interpreting information as natural frequencies
Experiment, 2x3x2, between subjects instead of probabilities (response mode) is
found to be useful for mitigating the dilution
Descriptive analysis, ANOVA
effect of irrelevant information in the context
of fraud risk assessments.
Eilifsen et al. (2020) 216 audit practitioners, Norway Exploration into perceptions of competencies
Semi-structured interviews and usefulness of ADA, as well as the
Questionnaire efficiency of ADA in the phases of an audit.
Findings show that the perceived acceptance
Qualitative and univariate quantitative
of ADA in the audit approach is higher than in
analysis
the external rules.
Emett et al. (2020) 71 auditors with peer review experience, External reviewers perceive lower audit quality
US in data analytics audit approaches compared
Experiments, 1x2, between subjects to traditional audit approaches. The results are
driven by external reviewers relying on the
tendency to judge audit quality based on the
perceived effort. Participants receiving an
audit execution primer do not rely on an effort
heuristic.
FRC (2017) Six largest UK audit firms Overview of the use of standard data analytics
Survey, questionnaire, meetings tools by type, drivers of the use of data
analytics, and required IT knowledge for their
use. Findings show that use of the tools
depends on the data environment of the audit
client, among other factors.
Gao et al. (2021) 17,000 individuals associated with 779 Data analytics competencies (of both frontline
audit offices (Big 4, mid-tier, small and back-office employees) improve audit
firms), US quality (using accounting restatements as a
Archival research (Compustat, data proxy). The effect is more pronounced for
analytic competencies listed on audit clients with complex business
Linkedln profiles) operations, clients that are more digitalized or
clients with complicated accounting estimates.
Hales et al. (2018) 1,265 companies, observations from Employee social media postings are useful to
2012-2015, Compustat, Glassdoor predict growth with regard to key income
(employee data), US statement information, transitory reporting
Archival research items, and earnings surprises.
Regressions
Hampton & 394 audit practitioners, Canada Overview of the motivation behind ADA use
Stratopoulos (2016) Survey, questionnaire and the importance of a supportive
environment. Results also give insight into the
Path models
effects of ADA expertise and diversity of
ADA tools available on audit opinion
confidence.
Holt & Loraas (2021) 120 auditors, US Auditors presented with varied data (e.g.,
Experiment, 2x2, between subjects varied visualization formats) provide more
conservative risk assessments and write-down
recommendations than those presented with
less ambiguous traditional evidence. The
46

conservative risk assessments and write-down
decisions are exacerbated when auditors are
placed in a high time budget pressure
environment.
Hosseinpour & Jans 14 auditors, Belgium, the Netherlands Only three out of six typical types of deviations
(2019) Interviews include a research instrument related to business processes are commonly
based on 18 procurement process traces used by auditors in their daily practice. This
might impact the acceptability of certain data
Qualitative and quantitative analysis
analysis techniques of auditors and audit
quality.
Jans et al. (2014) Procurement process, 31,817 payments, Process mining of event logs enables the
European Bank auditor to identify the most common steps in
Case study the procurement process and relevant
anomalous audit sequences, e.g., payments
Process mining
made without approval and violations of
segregation of duty controls.
Jans (2019) Seven process mining experts, Europe The study explores the choices made by
Case study-based interviews auditors during event log building when
applying process mining as an analytical
procedure. Choices related to the process
instance, activities, and attributes are
described and analyzed. Overall, auditors
have not yet transcended the traditional
auditing approach but use process mining for
enablement and partial automation.
Kend & Nguyen 20 auditors, regulators, audit committee Exploring the impact of advanced ADA on the
(2020) members and investors, Australia Australian audit and assurance market and
Semi-structured interviews how regulators need to respond to the impact
of ADA.
Kogan et al. (2014) Procurement data of a big healthcare This study examines disaggregated real-time
services firm, US data as primary driver of better estimations
Case study, simulated implementation of (prediction and detection of anomalies), e.g.,
a continuous auditing system, design amounts for the voucher payment process.
science The specific continuity equation used is a
second-order concern.
Estimations of continuity equations,
regressions
KPMG (2018) 250 corporate finance leaders, US, Data analytics provides benefits and is
Europe, Asia-Pacific perceived as a “must-have”. Data analytics
Survey capabilities such as visualization skills will be
important within the next two years.
Krieger et al. (2021) 16 auditors, data scientists, other experts, Exploring the early ADA adoption through the
Germany, Switzerland, US lens of process theory, ADA capabilities of
Semi-structured interviews the audit firm affect successful adoption,
identification of use-cases as a great
challenge.
47

Perols et al. (2017) 51 fraud firms and data from 15,934 non- Using advanced ADA methods (data pre-
fraud firm-years, 1998-2005, US processing methods) improves fraud
Experiment, comparison of prediction prediction when compared to the best current
models techniques.
Regressions and other methods
Rose et al. (2017) 127 senior auditors, US Auditors identify less crucial patterns in ADA
Experiment, 2x2, between subjects visualizations if they examine visualizations
before they have formed an initial expectation
ANOVA, ANCOVA (Analysis of
based on the results of traditional analytical
Covariance)
procedures.
Rose et al. (2021) 23 auditors, two Big 4 firms, US Visualizations that increase arousal (word
Experiment, 2x2, between subjects cloud/higher arousal vs. bar graph/lower
arousal) enhance auditors’ ability to recognize
Descriptive analysis, ANCOVA, simple
disconfirming evidence and incorporate this
effects
evidence in JDM. However, in the lower
arousal-setting auditors fail to consider
reliability of the underlying data while
making their judgments (suggested
misstatement amount).
Rozario (2019) 88 firms, 6,336 monthly Twitter Twitter-based information (number of tweets
observations from 2012-2017, that reflect past or future interest in purchase,
LikeFolio, Compustat, US and the total number of tweets that reflect
Archival research, simulated experiment positive and negative sentiment) is a predictor
of a firm’s sales and allows better error
detection.
Salijeni (2019), 25 auditors and regulators, EU countries Exploring how audit firms promote the
Salijeni et al. (2019), Semi-structured interviews relevance of ADA to relevant stakeholders,
Salijeni et al. (2021) especially focusing on the nature of the
relationship between auditors and their
clients, the consequences of the technology
for the conduct of audit engagements, for
communicating audit findings to the client,
training requirements, and the challenges
associated with embedding ADA in the audit
context.
Schreyer et al. (2019) 840,466 journal entry line items in SAP Deep autoencoder neural networks help to
ERP (Enterprise Resource Planning), detect local anomalies in journal entries of
two companies, Germany high relevance for a detailed follow-up audit.
Case Study
Autoencoder neural networks
Steinker et al. (2017) Sales data from Zalando’s data Weather data reduce sales forecast errors for
intelligence department and data from e-commerce (online) revenues.
20 weather stations from 2012-2013,
Germany
Archival research
Regressions, autoregressive integrated
moving average models
48

Tang (2018) 1,932 firms, 30,992 firm-quarter Twitter Twitter-based information (number of tweets
observations from 2012-2015, that reflect past or future purchase interest,
LikeFolio, Compustat, US and the total number of tweets that reflect
Archival research positive sentiment) is predictive of a firm’s
sales.
Walker & Brown- 27 internal and external auditors, US Exploring the emergence of data analytics
Liburd (2019) Semi-structured interviews through the lens of institutional theory,
especially focusing on factors influencing
audit firms’ decision to use data analytics, the
extent to which audit firms accept or reject
institutional demands, and audit firms’
experience of re-institutionalization related to
audit technologies.
Werner & Gehrke Journal entry data available in the ERP The developed system analyzes all recorded
(2019) system of a publicly listed company in transactions for indicators (related to mined
the manufacturing industry, Germany models) that signal weak or missing internal
Case study, design science controls. Based on this, the system generates
deficiency profiles. The system is successful
Process mining, factor analysis, research
in discovering internal control deficiencies
methodology is based on design science
providing a basis to conduct subsequent
research
substantive procedures.
Yoon (2016) One retail firm, observations from 2011, Weather variables are incrementally associated
private data, weather data with retail revenues (increased R2) and should
(wunderground.com), US be used as substantive analytical audit
Archival research evidence (development of an expectation).
Regressions and time series models
49

Appendix B. Future empirical research areas
Primary link to Selected future research Potential future research designs

the framework questions
Innovative What data sources, ADA audit tools, Archival research: to study the size of the audit
technology and embedded techniques are used firm and features of the audit client → use of
in current audit practice and what data sources, tools, and techniques
should be used? How does audit Survey and archival research: to study the use
practice apply these tools and tech- of general ADA audit tools and their
niques, i.e., sequentially vs. functionality as well as to explore potential
simultaneously? Which kind of data areas for generating additional data (such as
(aggregated vs. disaggregated) is drone and sensor applications).
used?
Experimental research: to study the degree of
data aggregation → cognitive processes,
audit quality, and audit efficiency
Design science research: to create artifacts that

contribute to the development, use, and
understanding of ADA in the audit domain
Case study: to apply different ADA audit tools

and techniques using data of a firm →
detection of anomalies, misstatements, risks,
audit quality
External rules Should educational standards require Experimental research: to study intellectual
certain competencies that are agility (e.g., the degree of cognitive closure),
necessary for the use of ADA? Is IT competencies → cognitive processes,
intellectual agility helpful and if so, audit quality, and audit efficiency
is this agility an unchangeable
personality trait or a trainable skill?
Should technical audit standards Interviews and experimental research: to

commit auditors to use ADA and explore the changes necessary in the audit
under which circumstances? How evidence standards
can ADA contribute to gathering Subsequent experiments: to investigate the
sufficient and appropriate audit influence of potential changes on cognitive
evidence? Is ADA a substitute for processes, audit quality, and audit efficiency.
test of details? Is ADA helpful to
evaluate the effectiveness of
50

internal controls and to what extent?
Are tests of controls dispensable?
Should the standard-setter follow a
principles- or rules-based approach?
Should quality standards refer to the Interviews and experimental research: to

use of ADA? explore the potential of ADA to support
auditors in conducting quality reviews
Subsequent experiments: to investigate the

influence of ADA on the effectiveness and
efficiency of quality reviews
Actors and their How should the audit team be Experimental research: to study the
organizations composed? composition of audit team → core categories
of cognitive processes, audit quality, and
audit efficiency
What structure should the audit Experimental research: to study the structure of
approach have? the audit approach (degree of formalization)
→ core categories of cognitive processes,
What drives the acceptance of the use Interviews: to explore the drivers of
of ADA within an audit firm? acceptance, such as perceived usefulness and
ease of use
How does the use of ADA influence Survey: to assess the public’s expectations of
the expectation gap? auditors’ responsibilities (e.g., level of
assurance) and current obligations as set out
in the audit standards
Does the use of ADA help to resolve Interviews: to explore whether and why ADA
disagreements between clients and can help resolve disagreements
auditors?
How does the use of ADA affect audit Archival research: to identify market
market concentration? concentration over time and drivers for
change
Audit process – What kind of data is useful to identify Interviews: to explore what kind of data is used
key steps and assess risks of material in audit practice and whether it is perceived
misstatements? to be useful
51

What methods are suitable to reduce Case study: to apply different methods to
false positives? identify false positives in historical
accounting data → audit quality
What kind of data is useful to detect Archival research: to explore the association
fraud, to assess an entity’s ability to between certain external data (e.g., weather
continue as a going concern, and to forecasts, social media data) and financial
audit accounting estimates? How statement items with the aim to improve the
can the auditor use this data as audit development of expected values and the risk
evidence? identification based on this data
Audit process – What problems come up while using Archival research: comparing data in databases
conceptual issues ADA as audit evidence? Is external with corresponding data in financial
data reliable? Is external data statements or other sources of origin in order
biased? How can auditors use to assess the reliability of data in databases
continuous data as audit evidence?
Is a data-driven audit process Interviews and experimental research: to

compatible with the ARM in its explore the components of the ARM in the
current form? context of ADA, focusing on potential new
components such as outlier risk and its
relation to the elements of the traditional
ARM
Subsequent experiments: to investigate which

components of the ARM and which linkage
(e.g., multiplicative) is best able to explain
overall audit risk evaluations and belief
adjustments during the course of an audit
Is the current assertion concept Interviews and experimental research: to

appropriate in the ADA context? explore the relationship between ADA and
Should the auditor consider other current assertions.
assertions? Subsequent experiments: to investigate how
the use of other assertions affects audit
quality
Is the current assurance concept Experimental research: to study ADA audit

appropriate in the ADA context? procedures vs. traditional audit procedures
How does the use of ADA affect the → comparison of the required level of
required level of assurance? assurance in percent.
52

How does the use of ADA affect the Interviews and experimental research: to
application of the concept of explore how to apply the concept of
materiality? materiality, e.g., when conducting process
mining, identifying outliers, or analyzing
disaggregated data
Subsequent experiments: to investigate the

impact of different materiality concepts on
audit quality
Cognitive processes What problem representations are Interviews: to explore what problem
– core categories appropriate in a ADA setting? representations auditors choose and how
auditors generate hypotheses in a ADA
What significance does the generation
setting: What information is necessary to
of hypotheses have in a ADA
generate an initial hypothesis? How does
setting?
hypothesis generation relate to information
How does belief-updating take place gathering in an ADA environment and
in a ADA setting? should these steps be reversed? Is there an
What are good indicators for increased risk to relying on an initially wrong
measuring audit quality in the ADA hypothesis in an ADA setting? Is this risk
context? related to the auditor’s intellectual agility?
What are the cognitive particularities of
using time series data or ambiguous data?
Experimental research: to explore problem

representation using different organizing
principles → generation of hypotheses →
search for information → auditor’s
conclusion → audit quality and audit
efficiency
Cognitive processes Does the current use of ADA lead to Interviews: to explore whether auditors are
– cognitive biases cognitive biases? Do auditors work aware of cognitive biases and how auditors
and affects backward to justify initial respond to them
hypotheses? Experimental research: to study different
What debiasing strategies should the visualization techniques, cognitive mindsets,
auditor use? How can auditors the scope of technical competencies,
overcome availability, confir- information presentation, and requirements
mation, dilution, and recency biases in the standards → core categories of
as well as inattentional blindness cognitive processes, cognitive biases, audit
and source credibility neglect in the quality, and audit efficiency
context of ADA?
53

How do affects influence JDM in the Experimental research: to study different
ADA context? What affects are affects (triggered by actors of the audit firm,
relevant? Do affects and cognitive the audit client, or the audit task) → core
distortions interact and if so, in what categories of cognitive processes, audit
way? quality, and audit efficiency
Cognitive processes How does the use of ADA affect the Experimental research: to study different forms
– environmental, auditor’s handling of aggressive of visualizations → assessment of the
personal, multi- client behavior and client pressure? reasonableness of management assumptions
person, and task
Is the auditor subject to system biases Experimental research: to study different ADA
factors
when using ADA? systems → core categories of cognitive
processes, audit quality, and audit efficiency
Does ADA usage risk deskilling and if Interviews and experimental research: to
so, how should this risk be handled? determine if there are features of a ADA
system that can promote the development of
the user’s expertise
Subsequent experiments: to investigate how

different features of ADA systems promote
the development of expertise
What factors influence the auditor's Interviews and experimental research: to

decision to rely on ADA? explore what factors influence an auditor’s
decision to rely on ADA
Subsequent experiments: to investigate

whether these factors affect auditor’s JDM
and audit quality
Which knowledge structures are Interviews and experimental research: to

preferable in the ADA context? explore the organizing dimensions used by
auditors while using different ADA tools and
techniques.
Subsequent experiment: to investigate whether

certain dimensions are preferable for higher
audit quality
What role does tolerance for Experimental research: to study the degree of
ambiguity play in the use of ADA? tolerance for ambiguity → core categories of
cognitive processes → audit quality, and
audit efficiency
54

What is the significance of Experimental research: to study metacognitive
metacognition while conducting skills → likelihood of generating and
analytical procedures in the ADA choosing correct explanations of correlations
context? identified in big data
How does the use of ADA influence Interviews: to explore how to use ADA in
the conduct of firms’ internal conducting internal quality controls.
quality control?
How does the complexity of the audit Experimental research: to study task
task influence the use of ADA? complexity, competencies → use of ADA →
55

SSRN Id3941961

Uploaded by

Copyright:

Available Formats

SSRN Id3941961

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SSRN Id3941961

Uploaded by

Copyright:

Available Formats

Empirical research frameworks in a changing world:

The case of audit data analytics

Freie Universitaet Berlin, School of Business and Economics

Electronic copy available at: https://ssrn.com/abstract=3941961

JEL Classifications: D91, M42, O30.

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

2.1 Empirical research framework - Basic design decisions

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

[Insert Figure 1 here]

2.2 Search for empirical studies

Electronic copy available at: https://ssrn.com/abstract=3941961

3 Empirical research framework - Categories and analysis

3.1 Diffusion of ADA

3.1.1 Innovative technology

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

3.1.2 External rules

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

3.1.3 Actors and their organizations

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

3.2 Actors reproducing rules and technologies

3.2.1 Audit process

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

3.2.2 Cognitive processes

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

3.2.3 Environmental, personal, multi-person and task factors

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Declarations of interest: none

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961

Electronic copy available at: https://ssrn.com/abstract=3941961