Assuring the quality of test and calibration results [5.

9]
7.5.1 Use of certified reference materials

An important way for methods to be calibrated by laboratories is the use of

certified
reference materials (CRM). A CRM is a sample for which the test results are firmly
established and agreed, ideally on an international basis. They are sold by some
national
standards bureaux and similar organisations and usually verified by highly
respected
reference laboratories or by interlaboratory calibration.
Acceptable procedures for certification of reference materials are detailed in ISO
Guide
35:1989, Guide to general and statistical principles for the certification of reference
materials. This
source also contains much information that can be equally applied in the
production of
in-house reference materials. The effective use of certified references and the
evaluation
of data generated by their use is covered by ISO Guide 33, Guide to the use of
certified
reference materials.
A laboratory which wishes to calibrate its methods can periodically check its
performance
by testing the CRM and so establish traceability. Laboratories which are able to
achieve
correct results with the CRM should, in theory, agree on any other test for the
same
parameters in the same matrix.
Page 47

In order to be effective, a CRM must be typical of the samples which the

laboratories are
testing on a routine basis. A method for effluents will, ideally, require a CRM which
is a
typical effluent.
Needless to say, CRMs must be stable and highly homogeneous as well as of
established
composition or properties. This is readily achieved in some areas, such as in the
chemical
analysis of alloys, the measurement of physical properties such as mass,
dimension, etc.,
and with some geological samples, but things are not so simple in other areas of
testing.
Materials testing, for example, often presents a particular difficulty since samples
are
usually totally destroyed in testing. The best that can be achieved here, therefore,
is to
have a large number of samples from the same source, for example cut from the
same
plate of metal or drawn from the same concrete mix, and to test a statistically
significant

number to arrive at an agreed figure for the whole batch.

Microbiology provides a different sort of problem since sample stability is virtually
impossible. However, in microbiology there are, at least, certified reference
cultures
which provide a definition of particular organisms so that laboratories can verify
that
their test systems are adequately selective. Some moves are in progress which it is
hoped
will lead to the ability to prepare quantitative microbiological references. These
are
generally based on the impregnation of cultures onto plastic supports of controlled
surface porosity.
Where certified reference materials are not available there are several alternative
strategies, but the main approach is participation in interlaboratory proficiency
exercisessee section 7.5.5. Such schemes, at least, give a laboratory a measure
of its
data relative to other similar laboratories and, if organised properly, provide a very
effective alternative to the use of certified references. Accreditation bodies will
generally
expect participation in appropriate schemes but, where certified references are
available,
these will be expected to be used as well.
Many basic test methods, especially in analytical chemistry, are intrinsically
traceable.
There is no need to have a certified reference for most titrations, for example.
Here,
traceability is provided via the calibrations of the balance and volumetric
apparatus. The
purist may argue that certification of the purity of the reagents which are weighed
or
measured is necessary but, provided the origin of the compounds is known and
they are
of known specification, it would be a harsh interpretation of the standard to insist
upon
this.

7.5.2 Use of spikes [5.9]

Spikes are widely used for method validation and calibration in chemistry and
microbiology. They provide a reasonable alternative to certified references, if the
spiking
material is adequately authenticated, ideally by certification of its purity. On the
face of it,
a spike has the advantage that the laboratory can spike into a matrix which is
absolutely
typical of its normal sample stream. The counter argument is to question whether
a
material spiked into the sample artificially is really present in the same distribution
and
speciation as the actual target. The strength of this argument depends on the
matrix. A

metal ion spiked into a water sample might well be regarded as a valid approach,
but a
pesticide spiked into a food sample may be questioned on the grounds that the
pesticide
in real samples was, perhaps, systemically absorbed by the crop used to make the
food
and so may be bound into the cell structure. However, in complex matrices the
spike
may be the only alternative, however imperfect it may be suspected to be.
A spike is generated by taking a real sample and adding a known amount of the
target in
question. Ideally, the base sample for the spike should have little or none of the
target
Page 48

present before spiking. If this is not possible, the spike level should be large
compared to
the natural level present. Of course, the natural level must be known in this
instance. The
spike must be thoroughly mixed and distributed homogeneously throughout the
matrix.
available.
The spike does not provide true traceability but it can be reasonably assumed that
laboratories which are able to demonstrate good recoveries of spikes have good
accuracy
and hence will tend to agree.
The use of spikes is especially important where laboratories are carrying out tests
in
complex matrices which may affect the results. Examples are water analysis where
matrix
effects are common and microbiology where components of the sample may well
affect
the viability of organisms. The spike, at the very least, demonstrates that the
laboratory
would detect the material or organism being sought if it were present.

7.5.3 References in calibration

The process of calibration involves the direct comparison of the item to be

calibrated
against a reference. It is, therefore, the reference itself which provides the
guarantee of
accuracy, and so it is critical that the reference itself is maintained and checked
regularly.
This will often only be possible by sending the reference for calibration. However,
in
many instances, the calibration laboratory can work with a hierarchy of standards
whereby a reference standard is maintained and used only for occasional checks
on
working standards. In calibration of balances, for example, this is common. The
laboratory will have a reference set of masses and, possibly, several working sets
which

are compared with the reference regularly. The reference is, perhaps, calibrated
externally
annually.

7.5.4 The use of quality control samples [5.9]

Quality control samples are used in exactly the same way as spikes and CRMs.
They are
merely samples for which the laboratory has established values and acceptance
limits.
They are tested along with unknown samples as a performance check. The
laboratory
may establish the values of the analytical quality control samples by repeated
testing, but
they should, ideally, be confirmed by at least two other laboratories.
If possible, quality control samples should be calibrated against CRMs. In this
instance
they become transfer standards, and the quality control sample provides
traceability. This
strategy is frequently adopted when expensive CRMs are needed since the
laboratory can
use the quality control sample routinely and check it only occasionally against the
expensive CRM.
ISO Guide 35 is a useful source of information on procedures for validating inhouse
quality control samples and confirming their homogeneity. Although, strictly
speaking,
the Guide is intended to refer to the production of CRMs, similar principles apply
to the
production of in-house reference materials for use as quality control samples.
As with the spikes, quality control samples which are not calibrated against CRMs
do not
provide traceability in themselves but demonstrate consistency of performance on
the
part of a laboratory. Such consistency, when combined with satisfactory results
from
interlaboratory exercises showing that the laboratory normally agrees with its
peers,
comes a very close second to establishing true traceability and is, in many
situations, the
only possible option.
Page 49

The laboratory will need to have a policy on the use of quality control samples and
for
evaluating and responding to quality control results. Guidance on this is given in
section
7.6.

7.5.5 The role of interlaboratory comparison [5.9]

Laboratories using spikes and other forms of analytical quality control samples are
effectively monitoring the consistency of their own performance. They should have
a

very good picture of their internal reproducibility. If they perform well on spikes
and on
any CRMs, they have also every reason to believe that their results are accurate.
Nonetheless, it is in the interest of any laboratory to test this assumption from time
to
time by exchanging samples with other laboratories and comparing results.
Such exercises are a very effective extension to the internal quality control
programme of
laboratories. They also provide an element of traceability when CRMs are not
available
since the more laboratories agree on results and the wider the range of samples on
which
they agree the more certain everyone can be of the accuracy of the collective
results. This
is further reinforced if agreement spans several analytical methods for the same
determinand.
Interlaboratory studies may be informal, in that a group of laboratories will
exchange
samples on an ad hoc basis, or may be formal exercises organised by a third party
who
circulates performance indicators. Irrespective of how it is done, the crucial part of
the
exercise is that the laboratory uses the data. This means reacting positively to any
results
which indicate that it is not performing as well as the other participants and
carrying out
remedial action.
Participation in appropriate interlaboratory proficiency schemes will normally be
required
by accreditation bodies, and some bodies actually operate their own schemes.
Although
interlaboratory comparison is listed in ISO 17025 as only one of several quality
maintenance options, most accreditation bodies will insist on its use wherever
possible. If
there are schemes available in a laboratorys sphere of activity or opportunities for
ad hoc
exchanges with other laboratories operating in the field, the accreditation
applicant will
have to provide good reasons to the accreditation body for not participating in
these
activities. The accreditation body simply sees inter-comparisons as the most
stringent test
of a laboratory and wants to see the laboratory subject itself to such a test.
The recently changed world security situation has resulted in severe difficulties in
shipping samples for proficiency testing across national boundaries, especially by
airfreight. This has, unfortunately, coincided with an increase in the insistence, by
accreditation bodies and particularly by the regional laboratory accreditation
conferences,
on proficiency testing as an activity for accredited laboratories. As a result it is
becoming

almost essential for any national accreditation body to ensure that adequate
proficiency
testing is available within the country before it can seek international recognition
through
MRAs and conference membership.
Accreditation will not normally be conditional upon any particular level of
performance
in interlaboratory comparison, but what will be required is for the laboratory to
have a
documented procedure for evaluating the results from its participation and for
responding to any problems revealed. There must also be records showing that the
results were evaluated and what action was taken to remedy problems.
The accreditation body will not withdraw accreditation on the basis of isolated
instances
of poor interlaboratory proficiency performance. However, if a laboratory is
consistently
Page 50

failing in proficiency testing and not taking effective corrective action,

accreditation will
certainly be jeopardised.
In the case of calibration laboratories, it is usual for accreditation bodies to
organise
interlaboratory comparisons. These are normally referred to as measurement
audits.
7.5.5.1 Procedure for evaluating performance in interlaboratory
comparisons
This activity must be formal, as noted above. A suitable approach is for the senior
laboratory staff to meet and to evaluate performance. The evaluation should
consider the
laboratorys known uncertainty of measurement. Whenever the laboratory
determines
that the result which it has returned differs from the expected result or from the
mean
obtained by other participating laboratories, then an investigation to establish the
reason
for the problem and the initiation of appropriate corrective action is called for,
together
with measures to check that the corrective action was effective. See section 4.8 for
discussion of corrective action.
The meeting should be minuted and the agreed action and its expected outcome,
recorded. Even if the meeting concludes that the laboratorys performance was
satisfactory, a record must be made.
The accreditation body will expect to see a file of the reports on interlaboratory
comparisons and, with each report, a record of evaluation of the data (the meeting
minutes) and a report on any corrective action.

7.5.6 Other methods for assuring confidence in results [5.9]

In the case of many methods, neither certified reference materials nor effective
spikes are
available. There may, however, be what are often referred to as consensus
standards,

recognised by all parties concerned. These would include many industry standards,
such
as those used in, for example petroleum source rock analysis or colour fastness
measurements. Such standards may not be traceable in a strict sense but are used
to
ensure consistency of data within the industry sector and hence form a basis for
agreement when testing against product quality standards.
There is another approach to testing that is also recognised as a means of
providing
confidence in results, a well established one in analytical chemistry:
determinations by
different methods which lead to comparable answers, which are generally
persuasive.
Repeat determinations are also used to provide confidence in results. Such
confidence
may be misplaced since errors may be repeated, especially systematic errors
resulting
from poor design of the system or errors in making up reagents. Where repeat
determinations can be valuable is if samples are retained over a relatively long
timescale
and then resubmitted, ideally blind to testing personnel, to check the consistency
of the
data.
Correlation of results from different characteristics of an item is also mentioned as
a
quality assurance method in ISO 17025. The extent to which this is relevant will
depend
not only on the type of testing being carried out but also on whether the client has
requested a range of tests rather than an isolated test. There is no explicit
requirement for
the laboratory to do additional tests to provide data for such correlations. Most
laboratories will scrutinise data in this way where they can, for example in water
analysis
to confirm pH, the hardness, alkalinity, conductivity and dissolved solids present a
consistent picture. This type of scrutiny should, however, be systematised and
there
should be guidelines in the methods documentation on the criteria to be used in
the
assessment so that it is made consistently.
Page 51

7.5.7 Reference methods

In the case of some determinands, different methods may give different results.
This may
arise because all of the available methods are more or less imperfect or because
the
method effectively defines what is measured.
BOD is an example of the latter, and BOD5 at 20oC with measurement by Winkler
titration is generally accepted as the definitive method. Many food proximate
analyses

also fall into the same category. Measurements of fat or fibre, for example, are
clearly
method-dependent since they do not measure precisely defined chemical species. If
fat
is determined by weighing the material extracted with pentane, for example, the
definition we are adopting for fat is that material which is extractable by pentane.
In these cases, the correct result is defined in terms of a reference method which
is
tightly specified, and traceability effectively means traceability to the reference
method.
Laboratories using methods other than the reference method should calibrate
them
against the reference method from time to time and would be under an obligation
to
demonstrate that any method which they choose to adopt gives data comparable to
that
from the reference method.
Even where targets are clearly defined, it may be necessary to agree on a
reference
method. This will arise when there are several methods which typically return
different
results.
Ideally, the technical problems implied by the inconsistency between methods
should be
resolved, the best method chosen and the rest discarded. Sometimes, however, it is
not
possible to come to a definitive answer on which method is technically superior
and,
especially where enforcement is the issue, one method is more or less arbitrarily
defined
as giving the correct result in order to solve the impasse.
Under these circumstances, interlaboratory calibrations are essential since the
method
defines the reference values and this is meaningless unless all participating
laboratories
are able to produce results which agree when all use the reference method.

7.6 Application of quality control and managing response

to results
[5.9]
Laboratories have traditionally approached quality control by including items of
known
properties in each test or calibration batch and evaluating the results against
defined
criteria to decide whether the data for the batch should be rejected or accepted.
This
approach has the virtue of simplicity and, provided the accept/reject criteria are
set
properly, it will defend the laboratorys contention that released data continues to
meet

defined performance characteristics.

However, if a laboratory only uses its quality control data in this fashion, it is
failing to
make full use of it. A laboratory might have a situation where all quality control
samples
are producing data which falls within the acceptance limits but always on one side,
for
example they may be consistently high relative to the expected value. This
situation bears
investigation since there should be a random scatter about the expected value.
This bias,
therefore, gives an early warning of a problem with the test or calibration system.
What is
really useful is that the problem has been detected before data is compromised.
Increasingly, assessors expect laboratories to make use of their quality control data
in this
fashion. Paragraph 5.9.1 of ISO 17025 makes explicit reference to recording data
in such
a way that trends can be detected, which strongly implies the use of control charts.
The
Page 52

next section gives an introduction to the use of control charts which might be
adopted
by a laboratory which is new to the area of statistical quality control.

7.6.1 Statistical quality control and preventive action [5.9, 4.12]

There should be an active co-ordination of quality control with a regular review of
performance, and records should be kept of the results of reviews and of any
action
taken in response. This should provide a mechanism for anticipating problems with
methods before they affect quality and as such is an important contribution to
preventive
action. See section 4.9 for further discussion of preventive action.
The management should collect together the results from quality control samples
for
each method and plot the data on a control chart. The most common control chart
format is the Shewhart, chart where the difference between the expected and
found
values for the quality control samples is plotted against time.
The Shewhart chart (see next page) gives a general visual indication when any
systematic
drift in the values returned by quality control samples is setting in. A method
which is
being used under control should show a random distribution of the actual values
about
the expected result, and any trends developing which suggest a bias should be
investigated.
It is usual to mark the control chart with a warning limit at two standard deviations
(2)
and an action limit at three standard deviations (3). The standard deviation is
generally

derived from the performance data determined at method validation.

The laboratory should, wherever possible, have a documented policy for deciding
when
the chart indicates a condition where the method should come under investigation,
and
this policy should be expressed quantitatively so that it will be applied consistently.
In
setting the rules for a particular method, the following widely accepted practice
should be
borne in mind.
Approximately 5% of results may be expected to fall outside the 2 warning
limits.
Any result outside the action limits requires investigation.
Two consecutive results outside the warning limits need an investigation.
A consistent run of successive results on the same side of the expected value
should
be investigated. It is widely accepted practice that a run of eight such successive
results triggers an investigation although many laboratories would feel the need to
respond rather earlier than this.
Consideration should be given to updating the control limits at regular intervals. If
the
method is consistently delivering data inside the current warning limits, then this
may
indicate that the uncertainty being achieved in practice is improving relative to the
data
generated at validation. Conversely, if the method is slipping out of control too
often,
this suggests that the validation data is presenting an optimistic picture.
Page 53

0.1

A common rule of thumb is to review the data every sixty points. If 1-6 (inclusive)
points
are found outside the 2 limits, then the indication is that the limits are
satisfactory. If
more points are found outside, then the limits are optimistic and either they should
be
revised or the method investigated in order to bring it back to a level the
performance
required. If no points are found outside 2, then new limits should be set reflecting
the
enhanced performance. Some laboratories take the view in this last case that they
will not
reduce the limits since this will result in increased rejection of data. In these
circumstances, the decision on whether to revise to tighter control limits will be
determined by whether the un-revised limits are acceptable in the sense that they
indicate
that the method is fit for purpose.

7.6.2 Frequency of quality control checks [5.9]

There is no simple answer to how frequently quality control items should be run.
The

trite answer is as often as necessary.

A general rule of thumb is that they should be included at a minimum rate of one
quality
control item in twenty but ideally one in ten. Experience of a method in a
particular
laboratory may indicate that more frequent checks are required.
In the case of methods which involve batch treatment of items, at least one quality
control should be present in each batch.
Some items can be tested or calibrated in duplicate as a check on the
reproducibility of
the method. It is far more useful, however, to expend the same effort in testing
another

Shewhart Control Chart

Action Limit +3

1.9
Warning Limit +2

1.6

Expected
value

1.0

Dateoftest
Warning Limit -2

0.4

Action Limit -3
X
X
X
X
X
X
X
X
XX
X
X
XX
X
XX
X
X
X
X
X
X
X
X
Page 54

quality control standard since this not only checks consistency but also gives
information
on overall error.
A method which can only be controlled by a high frequency of quality control
checks

should be looked at very carefully and seriously considered for replacement by a

more
stable method.