

Amazing
Audits
Evaluating an audit program’s
performance with a standardized
approach and the Kano model
by Lance B. Coleman Sr.

In 50 Words A QUALITY AUDIT program could

Or Less be considered a service for an organiza-
• A process and form al-
lows organizations to tion that’s provided by the quality depart-
evaluate the performance
of their audit programs. ment. Because of this, the Kano model can
• The evaluation’s score is
applied to the Kano mod- be the ideal lens for analyzing the perfor-
el, determining whether
the program satisfies the
mance of an organization’s quality audit
organization’s needs. program.
• The process allows or-
ganizations to track their In the 1980s, Noriaki Kano created the
audit programs’ status
over time and the effec- model as a theory of how to align product
tiveness of implemented
improvements. development with customer needs. Kano
 AUDITING

focuses on customer service and the ben-

efits of delivering exceptional value to the
customer, whether internal or external,
through service that delights the customer.
The Kano model recognizes four states:
nonperforming (failure), basic must haves
(cost of entry into the marketplace), per-
forming (more is better) and exciting or
delightful service (surprises and delights
the customer).1

September 2015 • QP X39

 The model (see Figure 1) stresses the pursuit of de- having the best plan in the world is worthless if that
lightful service delivery as a driver for continuous im- plan isn’t properly implemented.
provement and business success. It also can be used as The ability to support corporate and site goals is
a powerful visual for displaying progress in customer an often-overlooked segment of an internal audit pro-
satisfaction, and it reminds staff of the of the organiza- gram. It also should be seamlessly integrated into a
tion’s ultimate goal—delivering outstanding (delight- risk management program and play an integral role.
ful) performance. The next area of the form addressed reporting and
When I looked at this model, I asked myself: “What record keeping. For data to become information, it
would be attributes of a delightful audit program?” must be used, which was why data analysis was added
A delightful program would excel in monitoring to this area. Reporting, records and data analysis ac-
compliance, managing risk and driving continuous im- count for another 30% of the base score.
provement while the program itself continues to im- The reporting strategy for audit results was ad-
prove. To know whether you have an excellent audit dressed to ensure critical and major concerns were im-
program, an assessment must take place, but complet- mediately reported to the appropriate parties. Safety,
ing a quantifiable, accurate and objective assessment legal and functional issues are examples of findings
of an internal quality audit program is easier said than that might be reported to a wider group of managers.
done. The final 30% of the score relates to having adequate
I started by creating an evaluation form (see Figure resources, organizational structure, variability and
2, p. 43) to produce a ranking that could be incorpo- breadth of the audit program’s impact. Some questions
rated into the Kano model, which provides a needed that determine whether an audit program has the nec-
framework for evaluating the service of an internal au- essary resources are:
dit program. • Are there enough auditors to complete the number
of comprehensive audits required within the given
Evaluation forms timeframe?
The evaluation form assesses critical areas of an audit • Do auditors have the experience, skills and educa-
program, and it weights audit planning as the largest tion to evaluate the areas they’re auditing?
component, allotting it 40% of the base score. Planning • Is the required infrastructure—such as office space,
is a critical element in any endeavor, ensuring all as- computers and supplies—needed to manage the
pects of an organization that must be monitored are program in place?
consistently reviewed, and the evaluation form awards • Is there a budget for travel if off-site audits are re-
points for conducting the audits as scheduled. After all, quired?
• Are there authority and management support to con-
duct audits?
Kano model / FIGURE 1 After a base score has been attained, you will evalu-
ate how the audit program provides value beyond
Satisfaction verifying compliance and fulfilling requirements. This

+ gives the program credit for acting on opportunities for

improvement (see Table 1). If those opportunities had
potential financial savings attached to them, the audit
Delighters program received additional points.
(excitement factors) The next step is called the reality check (see Table
Primary satisfiers
2). Each major finding against the audit program found
Service Service
not fully during external audits, including findings previously
functional functional identified by the internal audit program that did not
have successful corrective action implemented, sub-
Must-be tracts points from the base score. Finally, there are
(basic requirements)
point deductions for findings related to issues not pre-
viously noted by the internal audit program and repeat

−
Dissatisfaction

40 QP • www.qualityprogress.com
 AUDITING

findings from the previous year. An organization also

should include repeat findings that resulted from an
Points added for value-add
external audit.
findings / TABLE 1
These points are added or subtracted from the base
Value add
score (see Tables 1 and 2) using a five-point rating
Points Description
scale:
+5 Observation that lead directly to $1,000+ lean savings.
• Five is very good or the best possible.
• Four is above average. +2 Observation that lead directly to $1-$1,000 lean savings.

• Three is minimally acceptable. +0.5 Opportunity for improvement that is implemented by area
management.
• Two needs improvement.
• One is very poor.
• Zero is nonexistent or almost nonexistent.
These definitions should be used to help score each Points deducted based on results
category. The sum of these scores is converted to a
100% scale where 80% or greater meets the goal, 60% is
of external audits / TABLE 2
minimally passing and 90% or greater (a minimum rat- Reality check
ing of three out of five in every category) is considered Points Description
excellent—or delightful on the Kano model.
−5 Each major finding against audit program from outside audit.
This assessment model allows an organization to
−3 Each major finding from outside audit that was previously
define each rating according to its guidelines. Simi- identified internally.
larly, the users decide what is adequate or more-than-
−2 Each major finding against quality management system from
adequate training for auditors, including the level of outside audit.
resources needed to carry out the mandates of their −1 Each minor finding against audit program from outside audit.
programs. After an organization defines its ratings, −1 Each repeat finding from previous year.
the model should be consistently applied to provide
results that can be analyzed and used as input to the
continuous improvement program.
After my audit program went through this rating
The audit cycle process, I looked at the score and found the initial rat-
An audit program manager (APM) develops plans ing reflected my first impression of the program when
based on organizational, site and audit program goals, I arrived at the site. The improvements I made during
which include various aspects of compliance, improve- my first year resulted in a better score, and it supported
ment and risk management. The APM may receive indi- my anecdotal and subjective opinion of the program’s
vidual audit plans for review and approval, but this de- status at that time.
pends on an organization’s policy and the experience The evaluation form’s elements were chosen be-
of the lead auditor. cause they’re as objective as possible and likely to be
The APM drives the audit’s execution phase by pro- consistent on a yearly basis.
viding policies, procedures, resources and credential-
ing requirements, and he or she will include this in the Other factors
auditors’ assignments. APMs receive reported audit Note that there are other important measures of an
results from lead auditors, and they analyze those find- audit program’s success not included in this model.
ings and report them to site management. This is because there are factors—beyond how robust
Responses to the audit are included as input in de- an audit program is—that might be involved in its im-
velopment of future plans. This entire cycle (see Online provement: for example, a yearly reduction in custom-
Figure 1, which can be found on this article’s webpage er audit findings.
at www.qualityprogress.com) is repeated with momen- Other possible reasons for improvement that are
tum from ongoing revisions of goals and objectives, unrelated to an audit program’s performance include:
including analysis of data provided by the audit results. different auditors, stricter clients, improvement in the

September 2015 • QP 41
 quality management system or luck of the draw (an au- components to meet your organization’s needs, and use
dit only takes a sample after all). it as an assessment of your assessment program. QP
Even other factors, whether they’re minor or major, REFERENCE
can contribute to a reduction of internal audit findings, 1. Kano Model Tutorial, ASQ.org, http://tinyurl.com/kanomodelasq.

and it’s difficult to consistently quantify monetary sav-

ings from findings.
LANCE B. COLEMAN SR. is a quality engineer and lean
These key performance indicators should, however, program coordinator at The Tech Group in Tempe,
improve yearly as your audit program’s rating and the AZ. Coleman holds a degree in electrical engineer-
ing technology from the Southern Polytechnical
program itself improve. Learn how to use this evalua- University in Marietta, GA. A senior member of ASQ,
Coleman is an ASQ-certified quality engineer, Six
tion tool, and read the sidebar “Case In Point,” which Sigma Green Belt, quality auditor, biomedical auditor
illustrates an example of an audit program before and and an Exemplar Global quality management system
principal auditor. Coleman is the author of Advanced Quality Auditing: An
after improvements were made using this model. You Auditor’s Review of Risk Management, Lean Improvement and Data Analysis
(ASQ Quality Press, 2015). He is an instructor for ASQ’s quality auditor
can use the evaluation form in Figure 2 as a guide. Ad-
certification refresher course and is the newsletter editor for ASQ’s Lean
just the ratings’ weights and add, change or remove Enterprise and Audit Divisions.

CASE IN POINT
Here is an example of how to assess an audit program based Before beginning a root cause and corrective action pro-
on existing records. Newly certified organization ABC was hav- cess analysis, the consultant rated ABC’s internal quality audit
ing trouble getting its internal audit program off the ground. program to provide an objective baseline for comparison after
A consultant was hired to improve the internal quality audit improvements were implemented. Figure 3 (p. 44) shows the
program after the organization received findings (one major and results of the assessment. A score of 60 is minimally acceptable,
one minor) against it in each of its two customer audits during 80 or better is the goal and a score of 90 or higher is delightful.
the preceding year. The auditors had received formal training, With a rating of 23% (based on the information from ABC’s
and there were no dedicated auditors; auditing was a second- records), ABC’s audit program was in the dissatisfaction quadrant
ary job function. of the Kano model (see Online Figure 2, which can be found on
After reviewing ABC’s records, the consultant noted the fol- this article’s webpage at www.qualityprogress.com).
lowing: The next year, the following changes were implemented:
1. The audit schedule for the previous year was not completed • Two new auditors were trained, and one worked on off shifts.
until June. • Two of the existing auditors received refresher training.
2. Only seven of 17 scheduled audits were conducted. • Portions of two audits were conducted during off shifts.
3. There was objective evidence that only one of two opportuni- • The standard operating procedure for audits was completely
ties cited was followed up on. rewritten.
4. The status of scheduled audits was not tracked. • Audit forms were revised, and auditors were trained to ad-
5. Audit forms were completed inconsistently from person to dress previously cited deficiencies.
person. These changes led to the following results from subsequent
6. Auditor meetings were not held. customer audits. There were no major audit findings against the
7. Audit results were not regularly reported. quality management system, and there wasn’t any nonconfor-
8. Training records were available only for four of six current mance issued against the audit program itself. Figure 4 (p. 45)
auditors. shows the results of the following year’s audit program evaluation.
9. Audit forms were poorly designed and did not logically flow. The improvements made to the audit program were substan-
10. Audit procedures were unclear and insufficiently detailed. tial, and as expected, they were reflected in the greatly improved
The consultant interviewed the existing auditors, and many score. The new rating was more than three times greater than the
expressed dissatisfaction with their levels of training. All original score—well above minimally acceptable requirements
confessed to occasional confusion about how to classify audit and within the margin of error for attaining the goal of 80%.
findings, how to complete various audit working papers and The score was in the must-be quadrant of the Kano model
when to pursue formal corrective action. They said it was due to (see Online Figure 2). There were still improvements that could
unclear work instructions and poorly designed forms. Some were be made, and some areas’ low scores could easily be used as
concerned with having too many audits to do along with their a roadmap for improvement and delightful performance in the
regular workloads. future. —L.C.

42 QP • www.qualityprogress.com
 AUDITING

Audit system evaluation form / FIGURE 2

Audit system evaluation

Date: 

  Planning (40%)
___  5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies).
___  5%—All scheduled audits were conducted.
___  5%—Percentage that were on time.
___  5%—All nonscheduled audits (not on the annual schedule, but scheduled to address a concern that was identified)
were conducted.
___  5%—Annual audit plan was approved by site management.
___  5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals.1
___  5%—Individual audit plans were approved by lead auditor or quality management with input from process owners.
___  5%—Audit program is well integrated within the risk management program.

  Reporting, records and analysis (30%)

Reporting—10%
___  Reported to site-level management.
___  Reported to quality manager.
Data analysis—10%
___  Reported data regularly analyzed and acted on.
Records—10%
___  Records maintained (audit reports).
___  Records maintained (training records).
___  Records maintained (quarterly or annual summary reports).

  Implementation and results (30%)

___  5%—Do we audit across all operating shifts?
___  5%—Is there an appropriate level of auditor training (requirements + actual)?
___  5%—Are an appropriate amount of audits dedicated to improvement activities?2
___  5%—Is a documented and structured method for evaluating the validity and classification of findings?
___  5%—Are there adequate resources?
___  5%—Are varying types of audits used, such as process, product, trace, system or element?

  Subtotal = potentially 100%

+   Value add percentage
–   Reality check percentage
 Total

Notes
1. ISO 9001:2008 and ISO 13485:2003 requires an audit program to monitor the effectiveness of a QMS, but it isn’t explained how you
know whether the QMS is effective. It’s not enough to simply state that a lack of findings demonstrates that the QMS is effective.
Metrics should be developed to assess the QMS, and the audit program should monitor those metrics in addition to compliance to
policies and procedures.
2. Improvement auditing is assessed against criteria that could be driven by compliance or a desired future state. It also could evaluate an
area for muda (nonvalue-add wastes) or monitor project results against predetermined milestones.

September 2015 • QP 43
 Organization ABC’s pre-improvement evaluation / FIGURE 3

Audit system evaluation

Date: mm/dd/yyyy

17%  Planning (40%)  17 of possible 40 total rating x 40% = (17/40) x 0.4 = 0.17 = 17%
  3   5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies). 
Yes, but were points deducted because the schedule wasn’t released until June.
  1    5%—All scheduled audits were conducted.  Only seven of 17 scheduled audits were conducted.
  0    5%—Percentage that were on time.
  5    5%—All nonscheduled audits (not on the annual schedule and scheduled to address a concern that was identified)
were conducted.  There were no nonscheduled audits (full credit).
  3    5%—Annual audit plan was approved by site management.  It was developed by the quality assurance manager.
  0    5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals. Not done.
  3    5%—Individual audit plans were approved by lead auditor or quality management with input from process owners. 
No input was provided from process owner.
  2    5%—Audit program is well integrated within the risk management program.  It’s informally tied to the risk management program.

13%  Reporting, records and analysis (30%)  13 of possible 30 total rating x 30% = (13/30) x 0.3 = 0.13 = 13%
Reporting—10%
  1    Reported to site-level management.  Only the number of audits conducted were reported during the management review.
  5    Reported to quality manager.  Results were reported to the process owner, and the quality manager signs each report for closure.
Data analysis—10%
  0    Reported data regularly analyzed and acted on.
Records—10%
  5    Records maintained (audit reports).  Hard and electronic copies of audit reports and schedule were maintained.
  2    Records maintained (training records).  Some training records were missing.
  0    Records maintained (quarterly/annual summary reports).  Not done.

10%  Implementation and results (30%)  10 of possible 30 total rating x 30% = (10/30) x 0.3 = 0.10 = 10%

  0    5%—Do we audit across all operating shifts?  Auditors work and audit only during the day shift.
  2   5%—Is there an appropriate level of auditor training (requirements + actual)? 
Auditors were trained, but requirements were not specified.
  0   5%—Are an appropriate amount of audits dedicated to improvement activities? 
Only compliance-based audits were conducted.
  3    5%—Is a documented and structured method for evaluating the validity and classification of findings?  They could be clearer.
  3    5%—Are there adequate resources?  There were adequate resources to implement the program.
  2    5%—Are varying types of audits used, such as process, product, trace, system or element?  Only element audits were done.

40%  Subtotal = potentially 100%

+   1%  Value add percentage  One point was added to score for an identified opportunity that was implemented.
– 18%  Reality check percentage Five points were deducted for one major audit program finding, -12 for six major QMS findings,
and -1 for one minor audit program finding.
23% Total

44 QP • www.qualityprogress.com
 AUDITING

Organization ABC’s post-improvement evaluation / FIGURE 4

Audit system evaluation

Date: mm/dd/yyyy

31%  Planning (40%)  31 of possible 40 total rating x 40% = (31/40) x 0.4 = 0.31 = 31%

  5   5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies).
  5    5%—All scheduled audits were conducted.
  4    5%—Percentage that were on time.  80% of audits conducted on time; including overflow from previous year.
  5    5%—All nonscheduled audits (not on the annual schedule and scheduled to address a concern that was identified)
were conducted.  There were no nonscheduled audits (full credit).
  4    5%—Annual audit plan was approved by site management.  Attachment in standard operating procedures was approved by
site management.
  1    5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals.
Improvement in this area was initiated.
  4    5%—Individual audit plans were approved by lead auditor or quality management with input from process owners. 
Now they are input from process owner.
  3    5%—Audit program is well integrated within the risk management program.  Findings are now classified by risk.

26%  Reporting, records and analysis (30%)  26 of possible 30 total rating x 30% = (26/30) x 0.3 = 0.26 = 26%

Reporting—10%
  4    Reported to site-level management.  They provided quarterly and annual summary reports, and input to management review.
  5    Reported to quality manager. Results were reported to process owner, and quality manager signed each report for closure.
Data analysis—10%
  2    Reported data regularly analyzed and acted on.  Improvement in this area was initiated.
Records—10%
  5    Records maintained (audit reports).  Hard and electronic copies of audit reports and schedule were maintained.
  5    Records maintained (training records).  All training records were current and available.
  5    Records maintained (quarterly/annual summary reports). 

16%  Implementation and results (30%)  16 out of possible 30 total rating x 30% = (16/30) x 0.3 = 0.16 = 16%

  2    5%—Do we audit across all operating shifts?  Parts of two audits conducted on off shift.
  4   5%—Is there an appropriate level of auditor training (requirements + actual)? 
  1   5%—Are an appropriate amount of audits dedicated to improvement activities?  Improvement in this area was initiated.
 3.5   5%—Is a documented and structured method for evaluating validity and classification of findings? 
 3.5   5%—Are there adequate resources?  There were adequate resources to implement the program.
  2    5%—Are varying types of audits used, such as process, product, trace, system or element?  Only element audits were done.

73%  Subtotal = potentially 100%

+   6%  Value add percentage  Six points were added to the score for identified opportunities that were implemented.
–   0%  Reality check percentage
79% Total

September 2015 • QP 45