GUID 3920

The Performance
Auditing Process

INTOSAI Guidances are issued

by the International
Organisation of Supreme Audit
Institutions, INTOSAI, as part of
the INTOSAI Framework of
Professional Pronouncements.
For more information visit
INTOSAI www.issai.org

Pre-IFPP document - this document was developed

before the creation of the INTOSAI Framework of
Professional Pronouncements (IFPP) in 2016. It
may therefore differ in formal purpose from more
recent INTOSAI Auditing Guidelines.
 INTOSAI

INTOSAI, 2019

1) Endorsed as ISSAI 3200 – Guidelines for the Performance Auditing

Process in 2016

2) With the establishment of the Intosai Framework of Professional

Pronouncements (IFPP), renamed and relabeled as GUID 3920 The
Performance Auditing Process - with editorial changes in 2019

GUID 3920 is available in all INTOSAI official languages: Arabic, English,

French, German and Spanish
TABLE OF CONTENTS

1. INTRODUCTION 4

2. PLANNING 6

SELECTION OF TOPICS 7

DESIGNING THE AUDIT 10

Managing audit risks 25

3. CONDUCTING 29
Evidence 30
Findings and conclusions 35
Reaching audit findings 36
Determining cause and effect of a finding 38
Developing conclusions after considering the findings 40
Managing audit risk in the conducting phase 42
Documentation during the conducting phase 43
Discussing the preliminary findings and conclusion
– internally and externally 43

4. REPORTING 45
Content of the report 46
Recommendations 52
Communicating with the audited entity 54
Distribution of the report 56

5. FOLLOW UP 60
1 INTRODUCTION

1) Professional standards and guidelines are essential for the credibility, quality
and professionalism of public-sector auditing. ISSAI 100 Fundamental
Principles of Public-Sector Auditing, amongst other things, defines the
purpose and authority of ISSAIs and the framework for public-sector
auditing. ISSAI 300 - Performance Audit Principles builds on and further
develops the fundamental principles of ISSAI 100 to suit the specific context
of performance auditing.

2) ISSAI 3000 is the Performance Audit Standard and should be read and
understood in conjunction with ISSAI 100 and ISSAI 300. It provides the
requirements for the professional practice of performance auditing followed
by explanations in order to enhance the clarity and readability of the
standard. ISSAI 3000 is the authoritative standard for performance auditing
and consequently each requirement must be complied with if an SAI chooses
to adopt it.

3) For each requirement set out in ISSAI 3000 supporting non-mandatory

guidelines are provided in GUID 3910 Central Concepts for Performance
Auditing, and in GUID 3920 The Performance Auditing Process.

4) GUID 3920 is intended to help the auditor interpret the requirements set
out in ISSAI 3000, and provide advice to the auditor on how to fulfil these
requirements and how to apply professional judgment.

5) The guideline is structured according to the different phases in the

performance audit process. The first section relates to planning the audit

4
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

– how to select audit topics and design the audit. The second section
relates to conducting the audit in order to obtain sufficient and appropriate
evidence to support the auditor’s findings and conclusions. The third section
relates to reporting – the format of the report, the report contents and its
distribution. The fourth section relates to follow-up of previous findings and
recommendations in performance audit reports, to identify and document
the impact of the audit and the progress made in addressing the problems.

Planning Conducting Reporting Follow up

6) GUID 3910 and GUID 3920 should be read together to get a deeper
understanding of how the central concepts are considered throughout the
audit process.

5
2 planning

Planning Conducting Reporting Follow up

7) This section contains requirements and guidance for planning performance

audits. The purpose of these requirements is to establish the overall
approach for the auditor to apply when planning the performance audit.
This section has two main parts. The first part is on selecting topics and
relates primarily to the SAI’s strategic planning process. The second part
of the planning section relates to the individual design of each audit,
focusing on what to audit, what criteria to apply and what methods of
data gathering and analysis to use.

6
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

SELECTION OF TOPICS

Requirements according to ISSAI 3000:

The auditor shall select audit topics through the SAI’s strategic planning process
by analysing potential topics and conducting research to identify risks and
problems. (ISSAI 3000/89)

The auditor shall select topics that are significant and auditable, and consistent
with the SAI’s mandate. (ISSAI 3000/90)

The auditor shall conduct the process of selecting audit topics with the aim
of maximising the expected impact of the audit while taking account of audit
capacities. (ISSAI 3000/91)

GUIDANCE

Selecting an audit topic as part of the strategic planning process

8) Determining which audits will be carried out is part of the SAI’s strategic
planning process. The SAI’s strategy provides the main direction for the work
of the SAI and therefore also that of its performance auditing function. The
strategy normally covers several years and guides the auditor in the selection
of topics, programmes or themes for audit. While the number of potential
topics, programmes and themes is usually high the SAI’s capacity is usually
limited.

Consequently, audit selection decisions must be taken with care.

9) The strategic planning process will normally result in an operational audit plan
for the SAI covering one or more years. Considering the dynamic nature of the
public sector and the changing priorities in public policies it is recommended
to revise the operational audit plan annually.

7
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

10) Some SAIs may choose topics based on strategic considerations regarding
the type of performance audit and reforms within the public sector. One
possible strategic choice is to contribute to the modernisation of the
government administration by focusing on auditing government programmes
with significant performance problems. Other SAIs may choose topics based
on other selection criteria, for example with regard to a specific type of
public sector activity audit such as the hospital sector or larger investment
projects. An alternative choice might be to simply focus on auditing individual
government agencies and their performance towards meeting objectives and
goals in relation to economy, efficiency and effectiveness.

11) The operational audit plan for the SAI will serve as a basis for operational
planning and resource allocation. The plan can list the audit areas and provide
a brief account of the possible problems, questions, and other arguments
supporting each one of them. The approach to selecting the audit topics to be
included in the audit plan may vary. Some SAIs have a bottom-up approach,
where the auditor participates in the selection process. Other SAIs have a top-
down approach, where the management selects audit topics and the auditor
does not take part in the selection process. Some SAIs have a mix of both
approaches.

Assessing potential audit topics in terms of risks, materiality and problems identified

12) The selection of audit topics can result from assessing risk, analysing problems
and considering materiality. Risks are the likelihood and impact of an event
with the potential to affect the achievement of an organisation’s objectives.
Materiality relates not only to financial aspects, but also social and/or political
ones, such as the number of people affected by a law or reform, transparency
and good governance.

13) In performance auditing, risks may involve areas of potential poor performance
that concerns citizens or have a great impact on specific groups of citizens. The
accumulation of such indicators or factors linked to an entity or a government
programme may represent an important signal to the auditor and can lead the
auditor to plan audits based on the risks or problems detected. Factors that
may indicate higher risk include:

8
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

a) Financial or budgetary amounts that are substantial, or significant

changes to the budget.

b) Areas traditionally prone to risk (for example IT systems, procurement,

technology, environmental issues and health).

c) New or urgent activities or when changes in conditions (e.g.

requirements and demands) are involved.

d) Management structures that are complex, with possible confusion

about responsibilities.

e) Lack of reliable, independent, and updated information on the economy,

efficiency or effectiveness of a government programme.

14) The analysis of potential topics should give consideration to maximizing the
expected impact of an audit. When analysing potential topics and conducting
research to identify risks and problems, the auditor is advised to consider the
following:

a) The greater the risk to performance in terms of economy, efficiency,

and effectiveness or public trust; the more important the problems
tend to be.

b) Adding value is about providing new knowledge and perspectives.

Greater added value can often be achieved by auditing policy fields
or subjects that have not previously been covered by audits or other
reviews.

Selecting audit topics that are auditable

15) Assessing auditability is an important requirement in selecting the audit topic.

At this stage, determining whether a topic is auditable or not depends on
whether the audit topics are within the audit mandate of the SAI and whether

9
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

the SAI has the audit capacities to conduct the audit. When designing the
audit, auditability will have to be considered yet again, in more detail (See
section on Designing the audit below).

DESIGNING THE AUDIT

Requirements according to ISSAI 3000:

The auditor shall plan the audit in a manner that contributes to a high-quality
audit that will be carried out in an economical, efficient, effective and timely
manner and in accordance with the principles of good project management.
(ISSAI 3000/96)

The auditor shall acquire substantive and methodological knowledge during

the planning phase. (ISSAI 3000/98)

The auditor shall elaborate the audit objective(s) in sufficient detail in order
to be clear about the questions that will be answered and to allow logical
development of the audit design. (ISSAI 3000/36)

If the audit objective(s) is formulated as audit questions and broken down into
sub- questions, then the auditor shall ensure that they are thematically related,
complementary, not overlapping and collectively exhaustive in addressing the
overall subject matter. (ISSAI 3000/37)

During planning, the auditor shall design the audit procedures to be used for
gathering sufficient and appropriate audit evidence that respond to the audit
objective(s). (ISSAI 3000/101)

The auditor shall submit the audit plan to the audit supervisor and SAI’s senior
management for approval. (ISSAI 3000/104)

The auditor shall actively manage audit risk to avoid the development of
incorrect or incomplete findings, conclusions, and recommendations, providing
unbalanced information or failing to add value. (ISSAI 3000/52)

10
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

GUIDANCE

16) After the SAI has chosen an audit topic, the auditor has to design the specific
audit. The auditor shall design the audit in a manner that contributes to
a high-quality audit that will be carried out in an economical, efficient,
effective and timely manner and in accordance with the principles of good
project management. A well thought-out design is in general indispensable in
performance auditing. This preliminary work is often called a pre-study.

17) The purpose of the design phase is to establish whether the conditions for
an audit exist and if so, to produce an audit proposal with a work plan and a
research design.

18) The design phase is normally carried out in a fairly short period. The design of
a specific topic can be conducted in less than a month, while a broader topic
can normally be conducted within three months. Generally, most of the time
allocated for the audit should be used for conducting the audit.

19) The auditor is advised to consider the needs and interests of the primary
intended users, including the responsible, parties when designing the audit.
The needs and interests of the users could influence the selection of audit
objectives and the types of analysis conducted by the audit team. Ultimately,
by taking into account the needs and interests of the primary intended users,
the auditor can ensure that the audit report is useful and understandable.

11
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

However, it is important that the auditor remains independent and focuses

on the main role of being a public sector auditor acting in the interest of the
citizens. The design phase should involve the elements presented in the box
below.

Box 1

Elements in the design phase

a) Assessing auditability

b) Understanding what is audited

c) Defining the audit objective(s) and audit questions

d) Defining the scope of the audit

e) Setting the audit criteria

f) Choosing methods to gather audit evidence

g) Good project management and review and approval of the plan by the SAI´s
management

o Managing audit risks

o Communicating in the planning phase

Assessing auditability

20) Assessing auditability is an important requirement in the design process. The

auditor must consider if conducting an audit is relevant and cost-effective. The
auditor may have to consider, for instance, whether there are criteria available
or whether the information or evidence required is likely to be available. Even
if the selected topic is consistent with the SAI’s strategy, the auditor might
observe during the design phase that the expected problem is already being

12
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

handled by the audited entity. Similar studies covering similar objectives may
already have been conducted by other institutions, or there are no relevant
criteria available and no reasonable basis exists for developing audit criteria.
Another reason could be that the information or evidence required is unlikely
to be available and cannot be obtained efficiently. On the other hand, the
absence of information and data can be a significant finding and the subject
of the audit itself and should not prevent the auditor from making further
inquiries. In such circumstances, it is important that the auditor informs the
management of the SAI of these concerns so that it can decide whether to
proceed or not.

Understanding what is audited

21) The aim at the beginning of the design phase is to develop a sound
understanding of the subject matter (‘what is audited’), and of risks and
challenges in the area. Performance audit is a learning process. Obtaining the
required knowledge is a continuous and cumulative process of gathering and
assessing information at all stages of the audit. Therefore it might be necessary
to gather further information and to test initial hypotheses in the design phase
once the audit topic has been chosen. This information will help the auditor
decide on the most relevant approach to the audit. It is important that the
auditor weighs the costs of obtaining information against the additional value
of the information to the audit. The information gathered in the planning
phase may make it necessary to adjust what is to be audited.

22) Sources of information for understanding what is audited may include:

a) enabling legislation and legislative speeches

b) ministerial statements, government submissions, and decisions

c) the audited entity’s risk profile

d) recent audit reports, working papers from other auditors, reviews,

evaluations, and inquiries

13
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

e) scientific studies and research (including those from other countries)

f) strategic and corporate plans, mission statements and annual reports

g) current and medium-term budgets

h) policy files, and management committee and board minutes

i) organisation charts, internal guidelines, and operating manuals

j) programme evaluation and internal audit plans and reports

k) viewpoints from experts in the field

l) discussions with the audited entity and key stakeholders

m) management information systems or other relevant information

systems

n) official statistics

o) reports from other SAIs

p) press coverage.

23) Past evaluations and audits are often a useful source of information. They can
help avoid unnecessary work in examining areas that have been under recent
scrutiny and highlight deficiencies that have not yet been remedied.

Defining the audit objective(s) and audit questions

24) The auditor shall set a clearly defined audit objective(s) that relates to the
principles of economy, efficiency or effectiveness (see ISSAI 3000/37). The
objective(s) determines the approach and the design of the audit.

14
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

25) The audit objective(s) can be thought of as the overall audit question concerning
the subject matter (for instance a government programme or activity) to
which the auditor seeks an answer. The audit objective therefore needs to be
framed in a way that allows a clear and unambiguous conclusion. The audit
objective(s) can be expressed in the form of one overall audit question which
is then broken down into more detailed/specific sub-questions.

26) The audit objective is often based on an overall perspective, that is, a top-
down perspective. It concentrates mainly on the requirements, intentions,
objectives and expectations of the legislature and central government.
However, it is also possible to add a client-oriented perspective, a focus on
service management, waiting time, and other issues relevant to the citizens
or users involved.

27) The audit objective(s) addressed by the performance auditor does not have to
be exclusively based on a retrospective (ex post) audit approach. The auditor
can undertake a study of on-going programmes, for example to study the level
of goal attainment, or to evaluate the progress made.

28) It is important that the audit objective(s) is based on rational and objective
considerations. In determining the audit objective(s), the auditor must
establish where the greatest problems or risks are, and where the audit can
add most value. To help define appropriate audit objective(s) the auditor can
conduct interviews with major stakeholders and experts, and analyse potential
problems from various viewpoints.

29) The audit objective(s) must give sufficient information to the audited
entity and other stakeholders about the focus of the audit. Well-defined
audit objective(s) relate to a single audited entity or an identifiable group
of government activities, systems, operations, programmes, activities or
organisations.

30) It is good practice to describe the audit objective(s) as simply as possible.

Presenting the audit objective(s) as clearly and concisely as possible prevents
the audit team from undertaking unnecessary or overly ambitious audit work.
The auditor is advised to avoid multiple objectives where more than one main
question is asked in order to be able to reach clear conclusions (see audit
scope below).

15
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Creating audit questions

31) It is good practice for the auditor to create audit questions that address the
audit objective(s). This will help define and structure the audit.

32) It is important for the audit questions to be thematically related,

complementary, not overlapping and collectively exhaustive in addressing
the audit objective(s). The aim is to cover all aspects of the audit objective
by specific audit questions. All terms employed in the questions need to be
clearly defined. The questions are stated in a neutral form, even if the auditor
expects to find problems in relation to the questions.

33) Audit questions may be analytical, normative or descriptive. Even if it is

advisable to formulate audit questions in a normative or analytical way,
adding descriptive questions can sometimes be useful in an audit, especially
when preparing an audit in an area where information on economy, efficiency
or effectiveness is lacking.

34) The formulation of audit questions is an iterative process in which the auditor
repeatedly specifies and refines the questions, taking account of known and
new information on the subject as well as the feasibility of obtaining answers.
During the planning stage, the purpose of formulating audit questions is
to systematically direct attention to what the auditor needs to know to
accomplish the audit objective. Audit questions may have to be adjusted, to
better reflect the subject matter as the auditor becomes more knowledgeable
during the audit (see Conducting in this ISSAI), but this should not be done
frequently. Since it is recommended that audit questions are communicated
to the audited entity, changing the audit questions during the course of an
audit may raise questions as to the professionalism, objectivity and fairness
of the audit.

Defining the scope of the audit

35) The audit objective(s), audit questions and scope are interrelated and need to
be considered together. Even minor changes in the objective(s) or the audit
questions may have a major impact on the general scope of the audit.

16
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

36) The scope defines the boundary of the audit. To define the scope, the auditor
needs to identify which entities are to be included in the audit or which
particular programme, or aspect of a programme, defines the boundary of
the audit. The auditor must also identify the period of time to be reviewed
and, if relevant, the locations to be included. To avoid an overly complex or
expensive audit, the audit scope can exclude certain activities or entities from
the audit, even if the activities or entities in principle would be relevant to the
audit objective.

37) It is good practice to discuss the audit scope with the audited entity at the
earliest opportunity. In some cases, it may also prove useful to explicitly
clarify what is not intended to be covered by the audit. This may help reduce
misconceptions or false expectations.

Setting the audit criteria

38) The auditor needs to establish suitable audit criteria, which correspond to
the audit objective(s) and questions (ISSAI 3000/47). Audit criteria are the
benchmarks or standard used to evaluate the subject matter in order to
determine whether a programme meets or exceeds expectations. The criteria
provide a basis for assessing the evidence, reaching findings and developing
conclusions on the audit objective(s).

39) The criteria can be qualitative or quantitative and define what the audited
entity will be assessed against. The audit criteria may be general or specific,
they may reflect a normative model for the subject matter under review, they
may represent best or good practice, or an expectation of “what should be”
according to laws, regulations or objectives. The audit criteria may also be
“what is expected”, according to scientific knowledge and best practice, or
“what could be” (given better conditions). The nature of the audit and the
audit questions determine which criteria are the most suitable.

40) The auditor can use many different sources to identify audit criteria, for
example:

a) Laws and regulations governing the operation of the audited entity

b) Political goals or statements by the legislature

17
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

c) Decisions made by the legislature or the executive

d) Key performance indicators set by the audited entity or the government

e) Detailed procedures for a function or activity

f) Standards from research, literature or professional and/or international

organisations

g) International benchmarks of good performance

h) Corresponding performance in the private sector

i) Benchmarks – same entity, different years; different entities same

activity

j) Planning documents, contracts and budgets from the audited entity

k) General management and subject-matter literature

l) Criteria used previously in similar audits or by other SAIs

m) Standards set by the auditor, possibly after consultation with subject

matter experts

n) Identification of what could be (given better conditions).

18
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

41) Sometimes audit criteria are easy to define, for example when the goals
set by the legislature or the government are clear, precise and relevant.
However, this is often not the case. The goals may be vaguely formulated,
conflicting or non-existent. This situation should not prevent the auditor
from conducting the audit. Under such conditions, the auditor may have to
establish criteria that reflect the ideal or expected result against which the
performance of the entity can be measured. One possibility is to set criteria
by allowing experts to answer questions such as ‘what ought the best
possible results be given the circumstances and according to best-known
comparable practice?’ If, on the other hand, the auditor uses performance
criteria or standards set by the audited entity, he or she needs to be cautious.
Meeting those standards does not necessarily equal good performance and
the auditor should be aware that the audited entity may set unreasonably
low standards in order to be sure to meet them.

Box 2

Tips for setting good audit criteria. Ensure that they are:

a) Relevant and logically or causally linked to the audit questions

b) Understandable, short and clear i.e. unambiguous and easy to comprehend

c) Complete, collectively exhaustive for each audit question – taken together,

they are sufficient to answer the audit questions

d) Objective, free from any bias

e) Testable so that it is possible to identify what procedures and evidence are

needed to provide an answer and to conclude against the criteria.

19
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

42) According to ISSAI 3000/51, the auditor shall, as part of planning or conducting
the audit, discuss the audit criteria with the audited entity and possibly with
the relevant stakeholders. Disagreement about criteria can then be identified,
discussed, and, hopefully, resolved at an early stage. This is especially important
where criteria are developed specifically for the engagement or where they
are not self-evident and may lead to a dispute with the audited entity.

43) It is important that the auditor listens to good arguments from the audited
entity when discussing the criteria, but at the same time is aware of their
potential interest in hiding their weaknesses. The facts and arguments
presented by the audited entity must be weighed against other relevant facts
and arguments (from other sources, experts etc.) and the auditor may accept
the audited entity’s perspective on the criteria after careful consideration.
However, the final decision on setting the criteria is the auditor’s and it is
important for the auditor to remain independent during this process.

Choosing methods to gather audit evidence

44) An important part of planning how to conduct the audit is to determine

the methods to be used to gather and analyse data. The audit objective(s),
audit questions, audit scope, and the audit criteria are the factors guiding
what evidence is needed and the methods most appropriate to obtain that
evidence.

45) During the design phase, the purpose of choosing methods is to systematically
focus on what the auditor needs to know to answer the audit questions and
address the criteria, and from where and how the auditor can obtain the
information. In this regard, it is advisable to conduct a pilot test in which the
auditor tests the data collection method. The aim is to adopt the best methods
and standards, but practical problems regarding the availability of data, or the
feasibility and cost of collecting it, may restrict the choice of methods. The
auditor may have to settle for the second-best solution.

46) If there is a problem with the availability of secondary data, or the quality
of the data is poor, the auditor could decide to collect primary data, by
developing questionnaires, statistical records, observations etc. While primary
data developed by the auditor is usually the most reliable, secondary data

20
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

collected and/or analysed by others (for example performance evaluation

reports, internal reports, etc.), can be an important source of information in
performance audits.

47) T he auditors may also have to consider changing the audit design. The auditor
has to decide which methods are appropriate to use in the audit, i.e. what
the advantages and disadvantages are and whether methods are too costly
to use compared with the expected outcome. In such situations the relevance
and value added by changing the design needs to be considered, and changes
need to be made as early as possible in the audit.

48) Performance audits can draw upon a large variety of data-gathering techniques
that are commonly used in the social sciences, such as surveys, interviews,
observations and the collection of administrative data and written documents.
Statistical sampling methods and surveys might allow for estimates to be made
for the whole population and case studies combined with other evidence
provide an opportunity for in-depth analysis.

49) Different types of audit evidence can be obtained by using different methods
of collecting data, as illustrated in the table below.

Table 1. Link between types of audit evidence and different methods

Audit evidence Methods of data collection

Interviews
Surveys, questionnaires
Testimonial evidence
Focus groups
Reference groups
Document review
File reviews
Documentary evidence
Using existing statistics
Using existing databases
Observation of people
Inspection of objects or processes
Physical evidence
Experiments, e.g. level of computer data
security
For instance: Quantitative data collection
methods. DEA analysis, regression analysis.
Analytical evidence Computations, comparisons, separation of
information into components, and rational
arguments

21
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

50) It is important for the auditor to establish an appropriate strategy for the audit
by combining study designs, methods and techniques in the audit to suit the
context, objective(s), questions, criteria and the auditor’s skills and resources,
as well as the availability of data.

An example of a design matrix

51) In designing the audit the auditor must link together the audit objective(s),
the audit scope and questions, the audit criteria and the methods for data
collection and analysis. There is no universally applicable model on how to
plan and design performance audits. The method presented below represents
good practice and is often applicable. It may not fit all audits but is useful
for complex engagements. In each case the auditor must therefore reflect
on whether the method presented below is suitable, or if there are better
alternatives.

52) The purpose of a design matrix is to clarify the feasibility of reaching a

conclusion on the audit objective, and to assure a logical chain of reasoning
and analysis from the audit objective to the audit criteria and the methods
used. The matrix helps the auditor to impose a logical, disciplined pattern on
the design of the study and to ensure that all aspects of an audit objective are
considered. Most importantly, the design matrix requires the auditor to clarify
at the planning stage what sources of evidence the audit criteria can be tested
against.

53) The figure shows how the audit objective(s) can be broken down into specific
audit questions and how each audit question is linked to audit criteria. When
drawing conclusions, the auditor makes use of professional judgment that
takes into account the different audit findings, the materiality of the findings
and the nature of the flaws revealed during the audit.

54) It is important that the auditor clearly outlines in the design matrix what kind
of analysis is needed to be able to obtain sufficient and appropriate audit
evidence in order to establish findings. This requires the auditor to describe
what kind of information and data are to be collected, the specific sources,
the techniques needed to gather data, and the kind of methods to be used to
analyse data.

22
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

55) As shown in the figure below, the expected audit findings are directly related
to the audit criteria. The auditor will have to assess whether the programme
or entity fulfils the criteria. If the audit shows that some of the criteria are met
while others are not, the auditor must use his or her professional judgment to
consider what the audit conclusion would be.

FIGURE 1: THE AUDIT DESIGN MATRIX (AN EXAMPLE)

AUDIT DESIGN

Audit objective

Audit questions

Audit criteria

Method

Audit evidence

Expected audit findings

(according to crieria)

Expected conclusions
(according to questions)

Expected conclusions
(according to objective)

OVERALL PROFESSIONAL
JUDGMENT IS REQUIRED

23
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Good project management and submitting the plan to supervisors and SAI
management

56) As performance auditing is time-consuming and costly, it is essential that the

audit is properly planned, and that the implementation of the plan can be
regularly monitored and corrective action can be taken when appropriate.
The audit plan should be written and documented and then submitted to the
SAI´s management for approval.

57) Collectively the audit team should have the necessary professional
competence to perform the audit. All team members need to understand
the meaning of the audit questions in the same way, the terms of reference
of the work assigned to them, and the nature of responsibilities required of
them by the applicable auditing standards. Nevertheless, one person, usually
the most experienced or the highest-ranking auditor, may be appointed
team leader. The team leader is responsible for executing the audit work,
as well as allocating tasks to the team members. The team leader ensures
high quality and timely production of the output by the audit team. It is
good practice to seek opportunities for staff development as part of the
audit work. An example could be to select a less experienced auditor as
team leader while ensuring support and coaching from a more experienced
auditor.

58) When making an audit plan, it is important to determine the timetable and
the resources needed. The auditor also has to consider if there is a need
to consult with internal or external experts (consultants, other auditors) in
order to enhance the quality of the audit. A milestone plan may help the
team break down the audit process into smaller parts. This makes it easier
for the team to assess how realistic the use of resources is compared to the
work needed. If the plan shows that the timelines are too tight, the team
may need to expand the timelines or consider which audit questions are the
most important ones, and spend time and resources accordingly. Sometimes
less important questions can be answered by using less time-consuming data
collection methods, such as using available secondary sources as opposed
to using questionnaires or interview data.

24
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

59) It is a good practice to document the use of budgeted resources and days
worked, as well as the milestones achieved. The latter can be done electronically
or documented in working papers along with important events in the audit.
Where more complex audits are concerned, the SAI may consider appointing
an experienced supervisor or a steering committee to guide the audit team
and to monitor the progress of the audit.

60) Audit supervisors provide guidance and direction to staff assigned to the audit
in order to address the audit objectives and follow applicable requirements.
At the same time they stay informed about significant problems encountered,
review the work performed, and provide effective on-the-job training. The
nature and extent of the supervision of staff and the review of audit work
may vary depending on a number of factors, such as the size of the audit
organisation, the significance of the work, and the experience of the staff.
Usually, it is also advisable to keep senior and top management regularly
informed about the progress of the audit and emerging findings, conclusions
and recommendations.

»» Managing audit risks

61) It is good practice to include a discussion of the specific audit risks and
how the auditor plans to mitigate them in the audit plan or pre-study.
Risk assessment can take many forms but may be done by addressing the
following questions:

a) Is there enough data available and is this data of good quality?

b) Does the audit team possess sufficient skills and knowledge for this
particular audit?

c) Are the time frames and resources (hours/funds) needed to conduct

the audit feasible?

d) Is the audit topic sensitive, highly visible or controversial? (political

sensitivity, media sensitivity, parliamentary sensitivity)

25
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

e) Is the audit and/or the subject matter very complex?

f) Is there a risk related to management integrity or entity relations?

62) If the audit risk is significant it may be necessary to develop strategies for
mitigating the risks and/or to modify the audit plan. The auditor can then
develop and modify the evidence collection strategy to lower the audit risk.
For example, it may be useful to consider:

a) Establishing a different staff mix – for example including more senior

staff

b) Using additional internal or external specialists

c) Adjusting the strategy and methods for data collection and analysis

d) Setting up specific communication arrangements with the audited

entity

e) Establishing specific quality control measures.

Communication in the planning phase

63) It is advisable to plan contacts with the audited entity and the relevant
stakeholders during the design phase and throughout the audit process in
order to keep them continuously informed of the audit’s progress.

64) Practices regarding communication may vary. Some SAIs prefer to give the
audited entity – especially senior management – detailed information on the
design of the study, since their early involvement can help to reassure the
audited entity and the responsible parties about the nature and scope of the
audit criteria. Other SAIs do not provide detailed information at this stage
and prefer to provide such information after the audit proposal has been
approved. Even without providing detailed information, it is generally good

26
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

practice to provide the audited entity with information on the assumptions

and reasons behind the decision to carry out the design phase. Preliminary
discussions with the audited entity are vital to inform them about the design,
what a possible audit could be about and why it may be undertaken.

65) Discussions with managers and staff at the audited entity are important
to gain basic knowledge of the audit area and its functions and conditions.
To avoid misunderstandings, it is advisable to inform the audited entity
involved about any such contacts. Dialogues with the audited entity might
further reorient the original objectives of the audit. In addition, it may also be
important to have discussions with the internal auditors and take advantage
of their experiences.

Box 3

The following topics may serve as examples for discussion with the audited entity
during the design phase:

a) whether the audit is requested by others, e.g. the legislature, or is at the

initiative of the SAI itself;

b) whether the audit is addressing a general risk, involves a strategic assessment

or whether it relates to economy, efficiency or effectiveness issues, and if so on
what grounds;

c) the purpose and the objectives of the pre-study;

d) the audit design, audit scope, timing and methodology

e) the audit criteria;

f) the kind of information the SAI may need to get from the audited entity at
this stage in order to build up knowledge, test potential designs, etc.

66) In addition to meetings and discussions with the audited entity, several
methods can be used to support the communication process in the design
phase:

27
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

a) Send a letter directly to the head of the audited entity. This will ensure
the proper presentation of the audit to senior management. A template
can be made to ensure that the information is presented in the same
way for all audits.

b) Produce a leaflet presenting the mandate and principles for performance

auditing as well as an outline of the audit process. This will facilitate the
understanding of what performance audit is all about and what the
audit process will include. It can be placed on the SAI’s website for a
general introduction to performance audit.

c) Identify contact people at the audited entity to enable the auditor to

have direct contact with the audited entity and make the audit process
run smoothly. Nevertheless, it is important to keep senior management
on both sides informed on important matters, something that may be
facilitated through a contact person.

67) It is the responsibility of the auditor to facilitate proper dialogue and

communication. However, if a disagreement occurs, it is important to
handle it in a professional and fair manner – listen carefully, focus on facts,
be objective and maintain one’s integrity.

28
3 CONDUCTING

Planning Conducting Reporting Follow up

68) This section contains requirements and guidelines for conducting

performance audits. The purpose of these requirements is to establish the
overall approach for the auditor to apply when conducting a performance
audit. The requirements relate firstly to obtaining sufficient and appropriate
audit evidence and secondly to using this evidence to answer the audit
objective and audit questions.

29
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

»» Evidence

Requirement according to ISSAI 3000:

The auditor shall obtain sufficient and appropriate audit evidence in order to
establish audit findings, reach conclusions in response to the audit objective(s)
and audit questions and issue recommendations when relevant and allowed by
the SAI’s mandate. (ISSAI 3000/106)

GUIDANCE

69) In order to answer the audit questions and be able to reach a conclusion on
the audit objective(s), the auditor has to gather the evidence needed. The
decisions on how to proceed with this data gathering process are generally
made when the audit is designed (see Planning above). Depending on how
detailed the general audit plan is, it can be necessary at the beginning of the
actual audit to elaborate further on where and how to collect the evidence
needed. It can be helpful to prepare detailed audit plans, if this has not been
done in the planning phase.

70) When the audit evidence is obtained, the auditor has to assess whether
the evidence is sufficient and appropriate. Based on this assessment, the
auditor has to decide if more or different evidence is needed.

How to obtain sufficient and appropriate evidence

71) In performance audits, evidence is rarely conclusive (yes/no or right/wrong).

More typically, audit evidence is persuasive (‘points towards the conclusion
that…’). When working in areas where evidence is persuasive rather than
conclusive, it can be useful to have discussions in the planning phase or at
the beginning of the conducting phase with the experts in the field about
the nature of the evidence to be obtained and the way in which it will be
analysed and interpreted by the auditor. This approach reduces the risk of
misunderstanding the evidence and may speed up the audit process. It is

30
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

also important that the auditor seek information from different sources,
since organisations, individuals in an organisation, experts, and interested
parties have different perspectives and arguments to put forward.

72) The aim of the planning phase is to properly and thoroughly plan the audit
in order to be able to apply the data collection methods described in the
planning documents. However, in the conducting phase, the auditor has to
continue identifying potential sources of information that could be used as
evidence. Not all situations can be foreseen during the planning phase, and
therefore the auditor may have to adjust the audit scope, questions, criteria
and methods for data collection and analysis during the conducting phase. It
is advisable to limit radical changes to the audit design unless unavoidable.
Material changes to the design are decided by management and the audited
entity needs to be informed about them.

73) The auditor also has to evaluate whether the lack of sufficient and appropriate
evidence is due to internal control deficiencies or other programme
weaknesses, and whether the lack of sufficient and appropriate evidence
could be the basis for audit findings.

Assessing whether the evidence is sufficient and appropriate

74) The concept of sufficient and appropriate evidence is integral to an audit

(See ISSAI 3000/106-111). In assessing evidence, the auditor has to evaluate
whether the evidence taken as a whole is sufficient and appropriate to
address the audit objectives and support findings and conclusions. Audit
objectives may vary widely, as may the level of work necessary to assess
the sufficiency and appropriateness of evidence to address the objectives.
The concepts of risk and materiality assist the auditor in assessing the audit
evidence (see also Audit risk and Materiality in GUID 3910).

31
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Box 4

Sufficiency

Is a measure of the quantity of evidence used to address the audit objectives and
support the audit findings and conclusions.

Appropriateness

Is a measure of the quality of the evidence that encompasses the relevance,

validity, and reliability of evidence used to address the audit objectives and
support findings and conclusions.

a) Relevance

o refers to the extent to which the evidence has a logical relationship with,
and importance to, the audit objective(s) and audit questions being addressed

b) Validity

o refers to the extent to which the evidence is a meaningful or reasonable

basis for measuring what is being evaluated. In other words, validity refers to the
extent to which the evidence represents what it is purported to represent

c) Reliability

o refers to the extent to which the audit evidence is supported by corroborating

data from a range of sources, or produces the same audit findings when tested
repeatedly.

75) As mentioned above, there are different types and sources of evidence
that the auditor may use. Each type of evidence has its own strengths
and weaknesses. The following contrasts are useful in assessing the
appropriateness of evidence:

a) Documentary evidence is more reliable than oral evidence, but the

reliability varies depending on the source and purpose of the document.

32
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

b) Testimonial evidence that is corroborated in writing is more reliable

than oral evidence alone.

c) Evidence based on many interviews together is more reliable than

evidence based on a single or a few interviews.

d) Testimonial evidence obtained under conditions in which people

may speak freely is more reliable than evidence obtained under
circumstances in which people may feel intimidated.

e) Evidence obtained from a knowledgeable, credible, and unbiased third

party is more reliable than evidence obtained from the management of
the audited entity or others who have a direct interest in the audited
entity.

f) Evidence obtained when internal control is effective is more reliable

than evidence obtained when internal control is weak or non-existent.

g) Evidence obtained through the auditor’s direct observation,

computation, and inspection is more reliable than evidence obtained
indirectly.

h) Original documents are more reliable than copied documents.

76) The following presumptions are useful in assessing the sufficiency of

evidence:

a) The greater the audit risk, the greater the quantity and quality of
evidence required.

b) Stronger evidence may allow less evidence to be used.

c) Having a large volume of audit evidence does not compensate for a lack
of relevance, validity, or reliability.

33
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

d) More evidence is normally necessary when the audited entity has

another opinion on the subject matter.

77) The auditor must determine the overall sufficiency and appropriateness
of evidence to provide a reasonable basis for the findings and conclusions,
within the context of the audit objectives. Professional judgment assists
the auditor in determining the sufficiency and appropriateness of evidence
as a whole (see also Professional judgment and scepticism in GUID 3910).
Interpreting, summarising, or analysing evidence is typically used in the
process of determining the sufficiency and appropriateness of evidence and
in reporting the results of the audit work. Where appropriate, the auditor
may use statistical methods to analyse and interpret evidence to assess its
sufficiency.

Box 5

Sufficient and appropriate evidence

a) Evidence is sufficient and appropriate when it provides a reasonable

basis for supporting the findings or conclusions within the context of the audit
objectives.

b) Evidence is not sufficient or appropriate when:

• using the evidence carries an unacceptably high risk that it could lead the
auditor to reach an incorrect or improper conclusion,

• the evidence has significant limitations, given the audit objectives and

• intended use of the evidence, or

• the evidence does not provide an adequate basis for addressing the audit
objectives or supporting the findings and conclusions. The auditor may not
use such evidence as support for findings and conclusions.

34
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

»» Findings and conclusions

Requirement according to ISSAI 3000:

The auditor shall analyse the collected information and ensure that the audit
findings are put in perspective and respond to the audit objective(s) and audit
questions; reformulating the audit objective(s) and audit questions as needed.
(ISSAI 3000/112)

GUIDANCE

78) Performance auditing involves a series of analytical processes that evolve

gradually through mutual interaction, allowing the questions and methods
employed to develop in depth and sophistication. The whole process is
closely linked to that of drafting the audit report. Reporting can be seen as
an essential part of the analytical process that culminates in answers to the
audit objective(s) and questions (see Reporting below).

79) The analytical steps to reach audit conclusions can be illustrated in this way:

1 Audit criteria - ‘what should be’

2 Audit evidence - ‘what is’

3 Audit findings - ‘what is’ compared with ‘what should be’

4 Determine the causes and effects of the finding

5 Develop audit conclusions according to ques�ons and objec�ve(s)

35
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

80) Step 1 is setting the audit criteria, which is done in the planning phase. Step
2 is the gathering of evidence and assessing whether these are sufficient
and appropriate. (See section on Evidence above.)

81) Step 3 is where the auditor uses the evidence to answer the audit objective(s)
and audit questions. When criteria are compared with the actual situation,
audit findings are generated.

82) Step 4: once the auditor has identified a deviation between ‘what should
be’ and ‘what is’, he or she is advised to determine, where possible, why the
deviation occurred (cause) and what the consequences (effects) of this are. .

83) In step 5, the auditor will reach a conclusion based on the findings.
Formulating conclusions may require a significant measure of the auditor’s
professional judgment and interpretation in order to answer the audit
questions and objective(s). It is necessary to consider the context and all
relevant arguments, and different perspectives before conclusions can be
drawn. The involvement of the SAI’s senior management is recommended
(see Quality Control in GUID 3910).

»» Reaching audit findings

84) Audit findings are the results of analysing and assessing specific evidence
and its relation to audit criteria. Hereafter the findings are used to answer
audit questions, which in turn serves as the basis for drawing conclusions
according to objective(s) (see the example of an audit design matrix under
Planning above). Audit findings normally contain the following elements:
criteria (‘what should be’) evidence (‘what is’), causes (‘why is there a
deviation from criteria’), and effects (‘what are the consequences’).

85) Meeting or exceeding the criteria may indicate “good practice” leading to
good performance. Failing to meet criteria would indicate that improvements
are needed. It is, however, unrealistic to expect that the audited entity’s
performance regarding economy, efficiency, and effectiveness will always
meet the criteria. It is important to appreciate that satisfactory performance
does not mean perfect performance, but is based on what a reasonable

36
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

person would expect, taking into account the audited entity’s circumstances.
Moreover, realistically, criteria cannot always be fully met. For instance, this
can be the case with some international agreements that can be difficult to
comply with fully in the short or medium term. ’Satisfactory performance’
may, in this regard, be understood as achieving what can realistically be
achieved in order to incrementally improve the situation towards meeting
the criteria. This means that regardless of whether the audited entity
meets the criteria or not, the auditor also has to consider materiality and
apply professional judgment in interpreting how this affects the entity’s
performance.

86) Most audits involve some type of analysis in order to understand or explain
what has been observed. When analysing information collected, it is
recommended that the auditor focus on the audit question and objective(s).
This will help to organise the data and also provide the focus for analysis.
A wide range of models or methods of analysis are used depending on
the objective of the audit and of the types of evidence used. Almost all
audits include different forms of document analysis. Depending on the
amount of documents and criteria that are analysed, the analysis can vary
in detail. Most audits also include interviews with the audited entity or
other stakeholders. These can be carried out in several ways depending on
the aim of the interview – structured interviews can serve as the basis for
more advanced analysis whereas less formal interviews can serve as the
basis for exploring the views of the audited entity. In addition, different
types of statistical methods can be used to analyse large quantities of
data – from basic to more advanced methods - depending on the type of
data used. In all audits discussion in the team and with colleagues and
management provides essential input, when findings and conclusions are
being assessed and drafted. Finally, discussions with the audited entity
and in some cases with focus groups may give the auditor valuable inputs
at this stage of the audit.

37
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Box 6

Examples of some different methods and models of analysis

a) Statistical analysis of data

b) Analysis of documents or physical evidence

c) Systematic analysis of interviews

d) Studies of documents or physical evidence

e) Discussions within the audit team, with experienced colleagues and

management

f) Discussions with the audited entity

g) Discussions with focus groups

87) The analysis may sometimes also require comparisons of findings, for
instance:

a) benchmarking of different institutions – the least successful institutions

can learn from the most successful institutions, and

b) comparing the audited area with a similar audit area in another country
or jurisdiction.

»» Determining cause and effect of a finding

88) While it is important to seek explanations for deviations from criteria, causes
must be presented with caution. They have to be supported by sufficient and
appropriate audit evidence. It is relevant to consider the audited entity’s
views on reasons for performance problems or weaknesses. If such views

38
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

are not based on sufficient and appropriate audit evidence, the auditor
cannot take for granted that they are relevant or correct. If the views of the
audited entity are not sufficiently supported by evidence, but reasonable
according to the auditor’s professional judgment, the auditor may choose to
present the view of the audited entity in the report.

89) The auditor is advised to identify the possible effects of the criteria not
being met. If possible, in identifying the effects, the actual situation must
be compared with the ideal condition where the criteria would have been
met. To a certain extent these possible effects would have been considered
at an earlier stage as a motivation for carrying out the audit of this particular
problem.

90) The effects could be identified either as what has already occurred or as
likely future impact based on logical reasoning. The nature of the findings
determines whether the auditor can present actual or potential effects.
Actual effects from past or current conditions help to demonstrate the
consequences and generally provide the reasons why corrective action is
needed. Potential effects are generally described as the logical consequences
that could follow when evidence does not meet the criterion. Potential
effects are to some degree speculative, so the auditor has to use them with
care, especially in the absence of any related evidence or observed past
effects.

39
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

91) It is important that the auditor carefully examines the relationship between
cause and effect and focuses on the possibility that other external factors
can influence the observed effect. It may be necessary to use enhanced
analytical techniques in order to answer questions on cause and effect.

Box 7

It is important to understand the nature of any relationships that may exist

between cause and effect. It is not always the case, for example, that poor funding
causes worse conditions. It might be due to the poor quality of care that funding
was reduced for a particular organisation. When scrutinizing data the auditor has
to remember that there are many reasons for relationships to exist:

a) There may be a direct cause-and-effect relationship. For instance, if a university

has a set intake each year, and increases its intake of part-time students, then it
must reduce its full-time intake.

b) There may be a reverse cause-and-effect relationship. For example, bad exam

results could be due to poor attendance but, equally, poor attendance could be
due to bad exam results.

c) The relationship may be a coincidence. For example, there may be a relationship

between the quality of health care in a local authority and exam results in that
same area, but it is hard to say that one causes the other.

d) There may be a confounding effect. For example, the relationship between

quality of health care and exam results could be due to effective use of resources
within the local authority, which may not have been considered as part of the
fieldwork.

»» Developing conclusions after considering the findings

92) Once the auditor has established the findings, determined why the criteria
are not being met (causes), and possible consequences (effects), the auditor
is ready to draw conclusions. Conclusions are statements inferred by the

40
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

auditor from those findings. Since performance audits point out deficiencies
in aspects of economy, efficiency and/or effectiveness, the conclusions have
to specify the reasons why aspects of economy, efficiency or effectiveness are
not fully meet.

93) Audit conclusions clarify and add meaning to specific findings in the report.
Conclusions present the auditor’s opinion and go beyond merely restating
the findings. Whereas the audit findings are identified by comparing ‘what
should be’ according to the criteria with the audit evidence (including
analytical evidence) on ‘what is’, the conclusions also reflect the auditor’s
explanations and views based on these findings. Conclusions might include
identifying a general topic or a certain pattern in the findings. An underlying
problem that explains the findings may also be identified.

94) The conclusions must flow logically from the findings, the performance
problems or weaknesses and their causes and effects. All analytical steps
taken beyond the findings will have to be clearly explained and justified.

95) When drawing conclusions it will often be necessary to revisit the data
analysis and the audit findings to be sure that the conclusions are based
on solid grounds. The analysis of data consists in combining results from
different types of sources. As mentioned before, there a different methods
which can be used in this analysis. In a properly conducted performance
audit, the arguments put forward are balanced against the best possible
counter arguments, and the various contrasting views are weighed against
each other. The conclusions have to be based on the objective(s) criteria,
evidence and findings.

96) When drawing conclusions, the auditor has to regularly test them against
the evidence base. Conclusions that are supported by different types of
evidence are likely to be sounder than those based upon only one source of
evidence.

97) In the process of developing conclusions it can be necessary for the auditor
to adjust or slightly change an audit question and on rare occasions even the
audit objective(s). It may be necessary to slightly adjust an audit question
when it becomes clear during the analysis that it will not be possible to

41
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

precisely answer the question from the evidence and findings obtained.
However, ideally such difficulties have been identified, and necessary
adjustments made, earlier in the audit process. If adjustments are needed
they have to be discussed and communicated both internally and with the
audited entity.

98) It is important for the auditor to be goal-oriented, to work systematically and

with due care and objectivity, and to exercise professional judgment when
analysing evidence. It is vital that the auditor adopts a critical approach
and maintains an objective distance from the information put forward.
At the same time, the auditor must be receptive to views and arguments.
The auditor must be able to see things from different perspectives and
maintain an open and objective attitude to various views and arguments.
An auditor who is not receptive may miss the best arguments. This also
underscores the importance of making rational assessments, in that the
auditor discounts his or her own personal preferences and those of others.
It is therefore important for the auditor’s involvement to be expressed in a
process of reflection and objective analysis rather than in a conviction that
certain standpoints are correct (See Professional judgment and scepticism
in GUID 3910).

»» Managing audit risk in the conducting phase

99) It is important to monitor audit risk and the planned mitigation strategies
throughout the audit, and adjust to changing circumstances when necessary.
Good planning will enable the auditor to manage audit risk when conducting
the audit, as the auditor will have planned for different eventualities and
scenarios. For example, if the planned data collection procedures do not
allow the team to collect sufficient evidence, the auditor needs to develop
an alternative plan for adjusting these procedures or, if necessary, adjust the
audit questions to be answered. Also, the auditor always needs to consider
whether the audit risks have changed in a way that can lead to inappropriate
conclusions, unbalanced information or not adding value. Proper quality
control procedures and supervision are important in this regard.

42
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

»» Documentation during the conducting phase

100) The auditor shall document the audit in a sufficient, complete and detailed
manner according to ISSAI 3000/86. Preparing audit documentation on
a timely basis helps to enhance the quality of the audit and facilitates
the effective review and evaluation of the audit evidence obtained and
conclusions reached, before the report is finalized. Because it is difficult to
reconstruct and recall specific activities related to gathering audit evidence
weeks after the work was actually performed, work needs to be documented
as the audit team completes it, in order to reduce the risk of inaccurate audit
documentation, improve audit quality, and improve engagement efficiency.

101) The nature and extent of audit documentation for a particular audit are
largely a matter of professional judgment, based on the unique circumstances
of each audit. However, an auditor will typically be expected to document
the following:

a) the objective (s), scope, and methodology of the audit; and

b) the work performed and evidence obtained to support significant

judgments and conclusions.

»» Discussing the preliminary findings and conclusion – internally

and externally

102) As the work continues, the draft report gradually takes shape. The notes and
observations are put into a structured order and, as internal and external
discussions progress, text is drafted, assessed and rewritten; details are
checked and conclusions are discussed. Communication is essential in the
analytical process because the auditor has to consider the context and all
relevant arguments, and different perspectives before conclusions can be
finally drawn. For this reason, the auditor needs to maintain an effective and
proper communication with the audited entity and relevant stakeholders.

43
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

103) Internal discussions with senior auditors and experienced colleagues can
help the auditor in the analytical process – in weighing findings and assessing
preliminary findings and conclusions.

104) Externally there is a need for exchange of information to discuss major

issues that have emerged during the audit. In this phase, it is necessary to
ensure that the factual basis of descriptions is accurate and fair, and that
the analyses are comprehensive and address the causes of the problems
identified. Various arguments need to be represented and findings put in
perspective. Meetings with the audited entity may serve to confirm facts
and to promote the development of audit findings and recommendations.

105) It is good practice to have meetings with senior managers or other

government officials. Another good practice is to carry out focus group
meetings, in which various stakeholders and experts are invited to discuss
preliminary findings, conclusions and recommendations. Being able to
discuss various issues with the participation of all the main stakeholders
will add value to the audit (See Communication in GUID 3910).

44
4 REPORTING

Planning Conducting Reporting Follow up

106) This section contains requirements and guidelines for reporting performance
audits. The purpose of the requirements is to establish the overall framework
for communicating the results of the performance audit. The requirements
relate to the form of the report, the report contents, and report issuance and
distribution.

107) The purpose of an audit report is to (1) communicate the results of audits to the
intended user(s); (2) make the results less susceptible to misunderstanding;
(3) make the results available to the public in order to create transparency,
unless specifically restricted by legislation; and (4) facilitate follow-up to
determine whether appropriate corrective actions have been taken.

45
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

»» Content of the report

Requirements according to ISSAI 3000:

The auditor shall provide audit reports, which are a) comprehensive, b)
convincing, c) timely, d) reader friendly, and e) balanced. (ISSAI 3000/116)

The auditor shall identify the audit criteria and their sources in the audit
report. (ISSAI 3000/122)

The auditor shall ensure that the findings clearly conclude against the audit
objective(s) and/or questions, or explain why this was not possible.
(ISSAI 3000/124)

GUIDANCE

108) Performance audit reports aim to improve knowledge and highlight

improvements needed. In a performance audit, the auditor reports on the
economy and efficiency with which resources are acquired and used, and
the effectiveness with which objectives are met. Such reports may vary
considerably in scope and nature, for example covering whether resources
have been applied in a sound manner, commenting on the impact of
policies and programmes and recommending changes designed to result in
improvements.

109) When writing the audit report it is vital that the audit team, supervisors
and quality control reviewers critically consider the conclusions in relation
to the audit findings, evidence, data, and audit criteria. Findings and
conclusions must be supported by sufficient and appropriate evidence.
Recommendations, if provided, must be linked to the findings and
conclusions. Proper procedures for clearance and fact validation with the
audited entity will also be important.

46
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Writing comprehensive reports

110) It is important that the report promotes adequate and correct understanding
of the matters and conditions on which the audit focused. To write a
comprehensive report, the auditor will typically include a description of the
audit objective(s) and the scope and methodology used for addressing the
audit objective(s) and audit questions. Readers need this information to
understand the purpose of the audit, the nature and extent of the audit work
performed, and the context and perspective regarding what is reported.

111) Readers also need to know if there are any significant limitations in audit
objective(s), scope, methodology, or data gathered, so that they can
reasonably interpret the findings, conclusions, and recommendations in the
report without being misled.

112) In the report, the auditor will typically identify significant assumptions
made in conducting the audit, and describe the methods and criteria used,
including their sources. The auditor has ultimate responsibility for defining
and explaining the criteria used in the audit report. (See Planning in this GUID
for more information on criteria.)

113) The auditor may provide background information to establish the context
for the overall message and to help the reader understand the findings and
significance of the issues discussed. Appropriate background information
may include information on how programmes and operations work, the
significance of programmes and operations, a description of the audited
entity’s responsibilities, and explanation of terms.

114) Most importantly, in order to write a comprehensive report, the auditor needs
to present sufficient and appropriate evidence to support the findings and
conclusions in relation to the audit objective(s).

Writing convincing reports

115) In a convincing report the audit findings and conclusions address the audit
objective(s) and questions, and are presented persuasively and objectively.
The report has a logical flow, with findings, conclusions and recommendations

47
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

clearly linked to the audit objective(s), audit questions, and audit criteria.
Furthermore, the conclusions and recommendations follow logically from the
audit findings and the facts and arguments presented. A convincing report
also needs to be accurate. An accurate report is fact-based, with a clear
statement of sources, methods and assumptions so that report users can
judge how much weight to give the evidence and conclusions reported. The
language and tone used is neutral, and the information presented is sufficient
to convince the readers as to the validity of the findings, the reasonableness
of the conclusions, and the benefits of implementing the recommendations.
Different perspectives, opinions and arguments are presented.

116) One way to help the auditor prepare convincing and accurate audit reports
is to use an engagement quality control reviewer. This is an experienced
auditor who is independent of the audit and checks that statements of facts,
figures, and dates are correctly reported, that the findings are adequately
supported by the evidence in the audit documentation, and that the
conclusions and recommendations flow logically from the evidence.

Writing timely reports

117) The report has to provide accessible, concise and up-to-date information,
which the government, the legislature, and audited entities can use for
improvements. To be of maximum use, the auditor’s goal is to provide
relevant evidence in time to respond to legitimate needs of the intended
users. Likewise, the evidence provided in the report is more helpful if it relates
to current issues. Therefore, the timely issuance of the report is important.
Some SAI’s control the timing of their work by setting specific tabling dates
to coincide with the sitting of the legislature. Therefore the tabling date of an
audit report is set in advance. In other cases, SAIs may have more flexibility to
determine deadlines, while considering the needs of intended users, and the
best timing for issuing the audit report.

118) During the audit, the auditor may provide interim reports of significant
matters to the audited entity, if allowed by the SAI’s mandate. Such
communication alerts the audited entity to matters needing immediate
attention and allows corrective action to be taken before the final report is
completed. (See GUID 3910 Communication.)

48
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Writing reader-friendly reports

119) To effectively add value, it is important that performance audit reports are
clear, concise, logical, and focused on the topic area. Reports are likely to have
the greatest impact on a wide audience when they are reader-friendly.

Meeting the audience’s needs

120) A key success factor for reader-friendly reporting is to determine the audience
and understand its needs. The primary audience for performance audit reports
is the legislature and government agencies. However, there are also other
stakeholders such as citizens, academia, the private sector and the media who
can all have an interest, but possibly a different focus, in the outcome of a
performance audit.

An effective report structure

121) At the outset of writing an audit report, it is important to determine a draft

structure to facilitate the organisation and flow of the text. An effective
structure enables the report to grab the reader’s attention, convey the
purposes of the audit, communicate complex issues, and provide clear
interpretation of the results. Using a “Dinner Party” approach can help the
auditor make a reader-friendly report structure. The Dinner Party approach is
analogous to a real dinner party situation where there is only a short amount
of time to hold fellow guests’ attention. The Dinner Party meeting takes place
after data collection and analysis and the aim is to produce crisp, interesting
report conclusions that can each be stated in 10-15 seconds, and to build up
more levels of detail from that basis.

49
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Box 8

Tips for designing a report which reads easily

a) Ensure the content of the report flows from the audit objective(s) and the
reader is provided with sufficient information to understand the topic.

b) Break up the text with the use of headings.

c) Be clear on the main point of each section and paragraph and how it relates to
the broader audit topic;

d) Design your report for easy reading, making it appear ordered and uncluttered.

e) Avoid the excessive use of cross-referencing and acronyms.

Clear writing

122) A reader-friendly report must be clear. In order to improve clarity:

a) avoid jargon. When technical, scholarly or foreign terms and
abbreviations are required they must be explained. It is helpful to
the reader if explanations are provided in a glossary or easy-to-find
footnotes;

b) avoid ambiguity;

c) use the same term consistently for a specific thought or object;

d) use active rather than passive voice;

e) be concise. Use short sections, paragraphs and sentences;

f) use examples that demonstrate audit findings and conclusions;

50
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

g) use visuals to draw attention to the main points. Use lists, tables,
diagrams, maps and other illustrations to present complex and large
amounts of data. These can often convey a message more effectively
than just text. However, keep tables and graphs simple. Make sure they
illustrate one idea only and that the reader will be able to understand
that idea immediately.

Writing balanced reports

123) The work underpinning performance audit reporting must be fair and
support the overall findings, conclusions and recommendations in order
to maximise impact. The auditor needs to explain the effects/impacts of
the problems in the audit report because it will allow the reader to better
understand the significance of the problem. This will in turn encourage
corrective action and lead to improvements by the audited entity.

124) In preparing a balanced and constructive report it is useful to:

a) resent findings objectively and fairly. Present and interpret facts

P
in neutral terms, avoiding biased information or language that can
generate defensiveness and opposition.

b) Present different perspectives and viewpoints. Where different

interpretations of the evidence can legitimately be made, these need
to be presented to ensure fairness and balance. By following the
underlying arguments, the reader will be able to understand the final
conclusions and recommendations better.

c) Be complete. A complete report includes both good and bad points

and gives credit where it is due. Including positive aspects may lead to
improved performance by other government organisations that read
the report. It is important that the report contains all the information

51
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

and arguments needed to satisfy the audit objective(s), and promote

adequate and correct understanding of the matters and conditions
reported. Facts must not be suppressed, and minor shortcomings not
exaggerated. Explanations, especially from the audited entity, must
always be sought and critically evaluated.

»» Recommendations

Requirement according to ISSAI 3000:

The auditor shall provide constructive recommendations that are likely to
contribute significantly to addressing the weaknesses or problems identified
by the audit, whenever relevant and allowed by the SAI’s mandate.
(ISSAI 3000/126)

GUIDANCE

125) Recommendations, where provided, aim to promote improvements by

lowering costs and simplifying administration, enhancing the quality and
volume of services, or improving effectiveness, impact or the benefits to
society. The auditor may recommend actions to correct deficiencies and
other findings identified during the audit and to improve programmes and
operations when the potential for improvement is substantiated by the
findings and conclusions reported.

52
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Box 9

Questions to consider when developing recommendations

a) What needs to be done and why?

b) Where does it need to be done?

c) Who is responsible for doing it?

d) Will the proposed actions remedy the problems observed?

e) Could the proposed actions have any negative effects?

126) It may be relevant to present the arguments for and against various alternative
proposals. By following the underlying arguments, the reader will be better
able to understand the final recommendations.

127) In order to be constructive, recommendations will typically:

a) be directed at resolving the causes of weaknesses or problems

identified;

b) be practical and add value;

c) be well-founded and flow logically from the findings and conclusions;

d) be phrased to avoid truisms or simply inverting the audit conclusions;

e) be neither too general nor too detailed. Recommendations that are

general will typically risk not adding value, while recommendations
that are too detailed would restrict the freedom of the audited entity;

53
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

f) be possible to implement without additional resources;

g) clearly state the actions recommended and who is responsible for

taking the actions;

h) be addressed to the entities with the responsibility and competence for

implementing them.

128) Effective recommendations encourage improvements in the conduct of

government programmes and operations. Recommendations are effective
when they are addressed to parties that have the authority to act and when the
recommended actions are specific, practical, cost effective, and measurable.

Box 10

Tips for generating recommendations

a) Think about potential recommendations early on in the audit process. Teams

are often expected to present the scope of potential recommendations at an
early stage.

b) Write the recommendations in a way that allows the auditor to evaluate

whether or not they have been implemented.

c) Where possible, work with the audited entity to identify the necessary changes
and ways of implementing them.

»» Communicating with the audited entity

Requirements according to ISSAI 3000:

The auditor shall give the audited entity the opportunity to comment on the
audit findings, conclusions and recommendations before the SAI issues its audit
report. (ISSAI 3000/129)

54
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

The auditor shall record the examination of the audited entity’s comments in
working papers, including the reasons for making changes to the audit report
or for rejecting comments received. (ISSAI 3000/130)

GUIDANCE

129) Giving the audited entity the opportunity to comment on the audit findings,
conclusions and recommendations, before publishing the report, helps to
ensure that the factual basis of descriptions in the report is accurate and fair
and that the analyses are comprehensive and address the cause of problems
identified. All of these issues need to be communicated to the responsible
authorities concerned by the audit.

130) Providing a draft report with findings for review and comment by audited
entities helps the auditor develop a report that is fair, complete, and objective.
Including the views of audited entities results in a report that presents not
only the auditor’s findings, conclusions, and recommendations, but also the
perspectives of the audited entity. This is particularly important in cases where
there are differences of opinion on significant facts presented in the report or
major disagreements on the appropriate course of action for improvement. It
is advisable to obtain the comments in writing.

131) Usually the SAI determines the amount of time given to the audited entity for
providing feedback, but care must be taken to ensure that there is sufficient
time.

Dealing with the comments received

132) All comments received need to be carefully considered. Where responses

provide new information, the auditor needs to assess this and be willing to
modify the draft report. All disagreements must be analysed in order for the
final report to be balanced and fair.

133) When the audited entities’ comments are insufficient to address the findings,
inconsistent or in conflict with the findings, conclusions, or recommendations
in the draft report, the auditor is advised to evaluate the validity of the

55
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

comments. If the auditor disagrees with significant comments, it is good

practice to explain the reasons for disagreement in a working paper. Conversely,
the auditor is advised to modify the report as necessary if the comments are
considered valid and supported with sufficient and appropriate evidence.

134) The responses need to be documented. It will be helpful to record the

examination of the feedback received in a working paper so that any changes
to the draft audit report, or reasons for not making changes, are documented.

135) At the end of the process it is advisable to keep the audited entities informed
about the procedures and timetable for the publication of the final report.

Referring the draft report to third parties

136) In order to ensure that the audit report is fair and balanced, it is good practice
to refer a draft report for comment to third parties concerned by the audit
as well as to the audited entity. Third parties are all those concerned by the
report, including all individuals and organisations referred to in the report. It is
good practice to give third parties the opportunity to comment on what is said
about them and their actions or views. While the third parties may be provided
with the entire draft for comments, the auditor needs to decide how much of
a stake third parties have in the subject matter. In some circumstances, the
auditor may choose to send third parties the whole report or major sections
of it, but often it will be appropriate only to send extracts.

»» Distribution of the report

Requirement according to ISSAI 3000:

The SAI shall make its audit reports widely accessible taking into consideration
regulations on confidential information. (ISSAI 3000/133)

56
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

GUIDANCE

Distributing reports to responsible parties, stakeholders and the public

137) It is recommended that SAIs decide on the method of distribution of reports

on the basis of their respective mandates. Each performance audit will
normally be published in a separate report, either in print or online, or both.
The reports must be distributed to the legislature and the responsible parties.
It is common practice to make reports accessible to the public and to other
interested stakeholders directly and through the media, unless prohibited by
legislation or regulations. It is an advantage if the reports are available for
public discussion and criticism.

138) If certain pertinent information is prohibited from public disclosure or is

excluded from a report due to the confidential or sensitive nature of the
information, the auditor should disclose in the report that certain information
has been omitted and give the reasons for that omission. Certain information
may be classified or may be otherwise prohibited from general disclosure
by legislation or regulations. In such circumstances, the auditor may issue
a separate, classified or limited-use report containing such information and
distribute the report only to those authorized by legislation or regulation to
receive it.

139) Additional circumstances associated with public safety, privacy, or security

concerns could also justify excluding certain information from a publicly
available or widely distributed report. For example, detailed information
related to computer security may be excluded from publicly available reports
because of the potential damage that could be caused by the misuse of
this information. In such circumstances, the auditor may make the data
anonymous or issue a limited-use report containing such information and
distribute the report only to those parties responsible for acting on the
auditor’s recommendations.

Results may be presented in different ways

140) The auditor is advised to use a form of the audit report that is appropriate for
the intended users, in writing or in some other retrievable form. For example,

57
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

the auditor may present audit reports using electronic media that are
accessible by all intended users. It is also good practice to publish a summary
of the report on the SAI’s website.

141) In addition to the published report (on paper or online), the auditor may
consider generating additional products to disseminate the findings in the
main report more widely:

a) It is recommended to provide the media with adequate and well-

balanced information, for instance in the form of press releases. This
may reduce the risk of the media misunderstanding or exaggerating
findings.

b) An executive summary of the final report, covering the most relevant

conclusions, could be made available to the general public on the SAI
web site.

c) Individual feedback reports may be issued to survey respondents to

show how they are performing compared to the sector benchmarks,
and to spread good practice.

d) Detailed data analysis may be published as technical annexes on the

web.

e) Other reports by consultants or academics on the same subject matter

may be placed in full on the web to give greater backing to the summary
provided in the main report. However, it is appropriate that this only
happens where it is felt these reports add substantial value and do not
conflict with the findings and the conclusions of the audit report.

142) The users’ needs will influence the form of the products, which may include
summaries, press releases or other presentation materials. Each product needs
to be written in a style tailored to its specific audience in order to have the
maximum impact. Preparing a communication plan can provide a structured
approach to thinking about how to reach different audiences effectively and

58
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

provide timely input to the decision making process (See Planning in this
GUID).

143) Whatever means are used to disseminate the message in the report, the
auditor needs to make sure that the messages are consistent throughout. The
auditor also needs to consider whether the products have been subject to
sufficient quality control.

144) In addition to written material, the auditor may use a range of means to
increase the influence of the audits by helping organisations to improve their
performance, and by spreading good practices and lessons learned across the
public sector. To do this, the auditor can use a variety of methods, such as
workshops with the audited entity to help stimulate and embed beneficial
change. Holding conferences is another effective way to reach practitioners
and promote discussion on important issues.

59
5 FOLLOW UP

Planning Conducting Reporting Follow up

145) The publication of the report is not the end of the auditing process. Beyond
publication there is follow-up on the impact of the audit. The aim of audit
reports is to influence the way in which services are designed and provided
to citizens, and to give recommendations to help deliver improvements in
the economy, efficiency and effectiveness of these services. This section
contains follow-up requirements and provides advice on how to do follow-
up of performance audit reports.

60
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

Requirements according to ISSAI 3000:

The auditor shall follow up, as appropriate, on previous audit findings and
recommendations and the SAI shall report to the legislature, if possible, on the
conclusions and impacts of all relevant corrective actions. (ISSAI 3000/136)

The auditor shall focus the follow-up on whether the audited entity has
adequately addressed the problems and remedied the underlying situation
after a reasonable period. (ISSAI 3000/139)

GUIDANCE

Why follow-up

146) Follow-up on the audit report is an important tool to strengthen the impact of
the audit and improve future audit work, and is therefore a valuable part of the
audit process. A follow-up process will facilitate the effective implementation
of audit findings and recommendations. It will also provide feedback to the
SAI, the legislature and the government on performance audit effectiveness
and the improvements made by the audited entity.

147) Following up on audit findings and recommendations may serve four main
purposes:

a) identify the extent to which audited entities have implemented changes

in response to audit findings and recommendations

b) determine the impacts which can be attributed to the audits

c) identify areas that would be useful to follow up in future work

d) evaluate the SAI’s performance. Follow-up provides a basis for

assessing and evaluating SAI performance and may contribute to better
knowledge and improved practices in the SAI. In this respect a follow-
up of audit reports is also a self-assessment tool.

61
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

The focus of the follow-up

148) Follow-up is typically done periodically as deemed appropriate by the SAI. The
priority of follow-up task are usually assessed as part of the overall the SAI’s
audit strategy. Sufficient time needs to be allowed for the audited entity to
implement appropriate action.

149) When conducting follow-up of audit reports, the aim is to determine whether
actions taken on findings and recommendations have remedied the underlying
conditions. This means that both positive and negative reactions regarding the
audit and the audit report need to be examined by the auditor. It is therefore
important to adopt an unbiased approach.

150) The impact of the audit may be identified through the effect of corrective
action taken by the responsible parties, or through the influence of the audit
findings and conclusions over governance, accountability, the understanding
of the problem addressed or the approach towards it.

151) When conducting follow-up of an audit report, the auditor needs to concentrate
on findings and recommendations that are still relevant at the time of the
follow-up. Insufficient or unsatisfactory action by the audited entity may call
for a further audit by the SAI.

How to do the follow-up

152) Different methods may be used to follow-up on the findings and

recommendations made.

a) Arrange a meeting with the responsible parties after a certain time

has elapsed to find out what actions have been taken to improve
performance and to check which recommendations have been
implemented.

b) Request the responsible parties to inform the SAI in writing about the
actions they have taken to address the problems presented in the audit
report.

62
GUID 3920 - THE PERFORMANCE AUDITING PROCESS

c) Use phone calls or limited field visits to collect information on the

actions taken by the audited entity.

d) Keep up to date on reactions from responsible parties, the legislature

and the media, and analyse whether problems identified have been
appropriately addressed or not.

e) Request financial audit to collect information about actions taken as

part of their audit procedures.

f) Carry out a follow-up audit, resulting in a new performance audit report.

153) The methods to apply will depend on the priorities established by the SAI
during the strategic and annual planning process for performance auditing.
They are also influenced by the importance of the identified problems,
the actions expected to be implemented, and the external interest in
information on the actions taken.

How to report the results of the follow-up

154) Whatever method is used, the results from the follow-up need to be
reported appropriately in order to provide feedback to the legislature. It is
good practice to report deficiencies and improvements identified in audit
follow-up to the responsible parties or the legislature.

155) Follow-up may be reported individually or as a consolidated report. If

several follow-ups are reported together, they may include an analysis of
different audits, possibly highlighting common trends and themes across
a number of reporting areas. Whatever the form, the follow-up reports
must be balanced and findings presented objectively and fairly.

63