DQA Policy

Guidance for Quality Assurance Staff in

Identifying and Reporting of Mandated
Quality Performance Indicators (QPIs) in
DE&S Contracts

Version 1. June 2009

Contents
The Need ....................................................................................... 3
Introduction...........................................................................................................3
Purpose ................................................................................................................3
References ...........................................................................................................3
The Requirement..................................................................................................3
The Quality Performance Indicators.....................................................................4
Measurement .......................................................................................................4
Background ..........................................................................................................4
Roles and Responsibilities ...................................................................................5

The Process .................................................................................. 5

Process Management ..........................................................................................5
QPI Establishment................................................................................................5

QPI Monitoring, Scoring and Reporting ..................................... 6

Mandated QPI Criteria..........................................................................................6
QPI Scoring and Reporting ..................................................................................6
The Supplier & Customer Performance Measurement (S&CPM) Process ..........7
Point of contact for further advice & guidance......................................................7

Annex A to Mandated QPI Guidance .......................................... 8

Mandated QPI Summary List ..................................................................... 8

Annex B to Mandated QPI Guidance .......................................... 9

QPI Scoring Customer on Supplier ........................................................ 9

Annex C to Mandated QPI Guidance ........................................ 14

Worked Example of a Contested Mandated QPI .................................... 14
Human Resource ...............................................................................................14
Supplier Stakeholder QPI Achieved ................................................................14
Customer Stakeholder Scored 6 out of 10 ......................................................14

Page 2 of 14

The Need
Introduction
1.

DE&S, its suppliers and their Trade Associations 1 wish to gain an

objective understanding on how the quality of the products delivered to
the front line is changing and how well the associated quality processes
are working. To this end a set of mandated Quality Performance
Indicators (QPIs) which are routinely called-up in DE&S contracts have
been identified to enable the consistent measurement of quality in the
acquisition process.

Purpose
2.

This document sets out the process to be followed by both DE&S and
Industry by which the mandated QPI data will be generated and
managed and reported. The process applies only to those direct
contracts let by DE&S where the overall value is > 5M vat ex. 2.

References
3.

The following standards are referred to in this guidance:

AQAP 2105 NATO

Requirements for Deliverable Quality Plans

AQAP 2110 NATO

Quality Assurance Requirements for Design, Development

and Production

Def Stan 05-61 Pt 1

Quality Assurance Procedural Requirements Pt 1

Concessions

Def Con 611

Government Furnished Assets

ISO 9001:2008

Quality Management Systems Requirements

The Requirement
4.

It is recognised by both DE&S and Industry that there is a need to

continually establish whether quality in the acquisition of defence
materiel and services is improving in line with the requirements of
AQAP 2110 section 8.1.

5.

The current performance criteria used by DE&S and Industry cover a

range of contractual activities which can include, but do not always
cover, quality. Therefore, as these performance criteria can vary from

From here on DE&S suppliers and their Trade Associations are known collectively as
Industry. This includes all those organisations at what ever level in the supply chain and
other bodies such as certification bodies.
2
The threshold of 5m is to maintain alignment with the DE&S Supplier and Customer
Performance Measurement (S&CPM) process operated by the DE&S Supplier Relations
Team.

Page 3 of 14

contract to contract depending upon the selection made, measurement

of any quality aspects has not been consistently achieved.
6.

By identifying a number of QPIs that can be monitored throughout the

life of a contract it is intended that this will establish a benchmark and
provide a basis for further consideration for improvement in the wider
acquisition of defence materiel or services.

7.

DE&S and Industry have a joint responsibility to ensure that continual

improvement is part of the acquisition process. Highlighting where the
process can be improved and the resulting action plans will result in
mutual benefit to both the DE&S and Industry.

8.

Each QPI is based upon the requirements of the contractual quality

standards i.e. Allied Quality Assurance Publications (AQAPs) and
Defence Standards. The measurement of contractual performance
against these standards is seeking to use information that should, as a
matter of course, have been generated as part of the normal contract
activities.

The Quality Performance Indicators

9.

Through the normal quality requirements contained in DE&S contracts

there are a number of discrete activities that are capable of being
reported on a periodic basis that over time will provide both DE&S &
Industry a better understanding of quality aspects.

Measurement
10.

Customer (DE&S) measurement of The Supplier:

Human Resources (Quality)
Quality Planning
Risk Management
Product Quality
Concessions

11.

Supplier measurement of the Customer (DE&S):

Government Furnished: Assets; Equipment; Information;
Facilities as agreed.
Response to concession applications
Human Resources (Quality)
Communication

12.

The respective Customer and Supplier teams may, in addition to these

QPIs, choose mutually agreed additional QPIs should they wish.

Background
13.

Through a series of joint DE&S/Industry workshops under the authority

of the Defence Industries Quality Forum (DIQF) and the Quality

Page 4 of 14

Assurance Consultative Groups (QACG) a set of QPIs were identified.

These were subsequently endorsed by the DIQF and QACG.
Roles and Responsibilities
14.

The roles and responsibilities of each of the participants are as

follows:a. DE&S Team Quality Assurance Focal Point
i.
Involve all project team stakeholders.
ii.
Include the mandated QPIs at the outset of the
contractual process
iii.
Identify the project QA or focal point for the QPI process.
iv.
Identify the management chain for QPI escalation.
v.
Agree the evidence details of the columns 1-2, 3-5 and 67.
vi.
Review the QPIs at the agreed timescale.
b. Supplier
i.
Include the QPIs in their Quality Plan.
ii.
Involve all Supplier stakeholders.
iii.
Identify the Supplier management chain for QPI
escalation.
iv.
Agree the evidence details for columns 1-2, 3-5 and 6-7.
v.
Review the QPIs at the agreed timescale.

The Process
Process Management
15.

The Quality Assurance Focal Points within the team are expected to
engage with other team members to ensure that the Mandated QPI
initiative is understood by all stakeholders in their respective teams
whether this is DE&S or Industry.

QPI Establishment
16.

Each QPI and associated criteria should be identified at the earliest

possible stage of the acquisition process (certainly no later than the
I.T.T/I.T.N stage) and should be capable of being tagged to the
specific Quality requirements contained in the relevant contract
documentation.
a. At Annex A is the list of mandated QPIs; a more detailed
breakdown identifying the origin of each QPI to the respective
Quality requirement documentation that will be included in the
contract is at Annex B.

Page 5 of 14

b. The formal quality plan to be produced by the supplier for the

contract should include a section, set-out in tabular form, which
lists the agreed QPIs and the action that will be taken by both
DE&S & Industry QA representatives in terms of monitoring and
reporting. Where any reference is made to
timescale/timeliness the details will need to be agreed through
consultation between DE&S and Supplier QA representatives
(All agreed timescales shall be documented)
c. Where no deliverable Quality Plan is to be requested, the
supplier will agree with the Customer QA representative the
approach to be taken to record the information relating to the
QPIs listed at Annex A for monitoring and reporting. {It is
recommended that the QPIs are set out in a table similar to that
shown at Annex B}.
d. The gradient approach (1-2, 3-5, 6-7) set out in columns e to g
on the QPI scoring matrix are provided to enable the
measurement of progress towards achieving the contractual
standard shown in column (i) and are intended to help both
DE&S and the Supplier in the monitoring of progress.
17.

Each QPI is to be capable of being associated with the respective

Quality requirements that will be called up in the proposed contract
requirements for AQAP 2105 NATO Requirements for Deliverable
Quality Plans.

QPI Monitoring, Scoring and Reporting

Mandated QPI Criteria
18.

The respective Supplier & DE&S QA representatives will need to

establish a regular dialogue and agree a plan for monitoring the agreed
QPIs during the life of the contract.

QPI Scoring and Reporting

19.

At Annex B a scoring matrix is provided (columns e to i), for the

purposes of measuring contract performance against the QPI criteria,
the definition provided in column (j) will determine whether the specific
QPI has been achieved or not achieved (A or NA).

20.

Where any of the QPIs have several components that make up the
overall requirement e.g. Human Resources Competence, Awareness
& Training which has five, the lowest score against each element will
determine whether the QPI has been achieved or not achieved.

Page 6 of 14

The Supplier & Customer Performance Measurement (S&CPM) Process

21.

Rather than introduce a new reporting process for capturing QPI

information the decision was made to use the existing S&CPM process
operated by the DE&S Supplier Relations Team.

22.

The agreed QPIs documented in the Contract Quality Plan (or in a

separate document where there is no deliverable quality plan) will also
need to be included in the S&CPM Performance Review
Agreement (PRA) document.

23.

Details of the S&CPM process can be found on the MOD Internet site
link here
http://www.MOD.uk/DefenceInternet/MicroSite/DES/OurPublications/Su
pplierAndCustomerPerformanceMeasurementProcessscpm.htm

Point of contact for further advice & guidance

24.

Any further information or queries relating to mandated QPIs should be

directed to:
DE&S
Defence Quality Assurance (Policy) 030 679 32681
Supplier Relations Team 030 679 34279, or
Via your QACG representative
Industry
Via your DIQF Trade Association representative

Page 7 of 14

Annex A to
Mandated QPI Guidance
Mandated QPI Summary List
QPI Ref
Area
Human Resource
1
1a.
1b.
1c.
1d.
1e.
Quality Planning
2
2a.
2b.

QPI Specific
Decisions/Actions related to resource.
Competence.
Training.
Training effectiveness evaluation.
Training records.
Quality objectives established.
Verification, monitoring, inspection and
test activities determined.
Records to provide evidence
Review of QMS
Records from Management Reviews.
Risk planning
Quality Escapes

2c.
2d.
2e.
Risk Management
Product Quality
Concessions

3.
4.
5.

Clear benefit to the Authority.

Minor concessions recorded/controlled.
Concession records available.

5a.
5b.
5c.
Government Furnished
Assets

6.
6a.
6b.

MOD Response to
Concessions
DE&S Competence
Communication

7.
8.
9.
9a.
9b.
9c.
9d.

Delivered in accordance with plan.

Fit for purpose
Within agreed timescale.
DE&S team empowerment
DE&S points of contact provision.
DE&S feedback to support Management
Reviews.
DE&S response for information within
agreed timescale.
DE&S feedback on quality issues within
agreed timescale.

Page 8 of 14

Annex B to
Mandated QPI Guidance
QPI Scoring Customer on Supplier

(a)
AREA /
REFERENCE
Human
Resource.
AQAP 2110 para
5.1(e).
AQAP 2105 para
4.5

AQAP 2110 para

6.2.2.
AQAP 2105 para
4.5

(b)
QPI SPECIFIC

CRITERIA

ACHIEVED

NOT ACHIEVED

FOR S&CPM PROCESS PURPOSES:(d)

(e)
1-2

(f)
3-5

(g)
6-7

(h)

(i)
8-10

(j)
Remarks

Management
Commitment

Management reviews shall

include any decisions and
actions related to resource
needs.

Quality resource needs for

delivery of contract reviewed
by top management during
management review and
where deficiencies identified,
appropriate action taken for
gap to be closed by next
S&CPM assessment.

No Evidence
that Quality
resource
needs
reviewed and
actions
identified.

Insufficient
Evidence that
Quality
resource needs
reviewed and
actions
identified.

Some evidence
that quality
resource needs
reviewed and
actions
identified and
mitigated.

Quality
resource
needs
effectively
reviewed and
deficiencies
mitigated.

Competence,
Awareness &
Training

Necessary competence for

personnel performing work
affecting product quality.

Necessary competence of
quality resource reviewed.

< 80

80 - 90

90 - 95

>95

%age of quality personnel's

competence checked in last year.

> 20%

20%-10%

10%-5%

below 5%

>10%

10% - 6%

6% - 3%

< 3%

of identified competence shortfall

does not have a closure plan
outstanding evaluations

Training or actions taken to

satisfy need
Effectiveness of actions
taken evaluated
Records of education,
training, skills and
experience maintained.

No Records of Little Evidence Some Records

Evidence
of Records.
of Evidence

> 95%
Records
maintained

Quality Planning.
AQAP 2110 para
5.4.1
AQAP 2105 para
4.6.

Quality
Objectives

Management shall ensure

that quality objectives,
including those needed to
meet requirements for
product are established at
relevant functions and levels
within the organisation.

No evidence
that Quality
Objectives are
established

Page 9 of 14

Insufficient
evidence that
Quality
Objectives are
established

Some evidence
that Quality
Objectives are
established

In over 95% of
areas Quality
Objectives
established &
reviewed
annually or at
the start of
each phase.

Quality objectives are established in

all functions and at all levels and
reviewed at least annually or the start
of each phase.
Note: Quality objectives shall be
measurable and consistent with the
quality policy.

(a)
AREA /
REFERENCE
AQAP 2110 para
7.1
AQAP
2105 para 4.6.1

AQAP 2110 para

5.1.(d)
AQAP 2105 para
4.6

(b)
QPI SPECIFIC
Product
realisation
planning

Management
Review

CRITERIA

(d)

(e)
1-2

(f)
3-5

(g)
6-7

(h)

(i)
8-10

(j)
Remarks

The organisation shall

determine the following:

Required verification,
monitoring, inspection and
test activities specific to the
product and the criteria for
product acceptance.

No Verification
Activities
related to
product

Insufficient
evidence of
Verification
Activities
related to
product

Records needed to provide

evidence that the realisation
processes and the resulting
product meet requirements.

No Records of Little evidence Some evidence

evidence
of Records.
of Records

Management shall review

the organisation's QMS at
planned intervals to ensure:

No evidence of Little evidence Some evidence

Reviews of
of Reviews of of Regular
QMS
QMS
Review of QMS

Records from management

reviews are maintained.

No records
maintained.

Insufficient
records
maintained.

Page 10 of 14

Evidence that
some
Verification
Activities
related to
product

Evidence that
some records
are maintained.

Over 95% of Acceptance plan covers verification,

product
monitoring, inspection and test
requirements activities.
have
verification,
monitoring &
inspection
activities as
required in the
Acceptance
plan.
> 95%
Records
maintained

> 95% of QMS Reviewed at Management Review

reviewed.
where the QMS' continuing suitability,
adequacy and effectiveness also The
need for changes to the QMS,
including quality policy and objectives
are covered.
All
management
reviews are
accurately
recorded in a
timely manner

(a)
(b)

AREA /
QPI SPECIFIC
CRITERIA
REFERENCE
Risk
Management.
AQAP 2110 para Planning
The Supplier shall provide
5.4
AQAP
objective evidence that risks
2105 para 4.7
are considered during
planning, including but not
limited to Risk Identification,
Risk Analysis, Risk Control
and Risk Mitigation.

Product quality. Quality escapes

AQAP 2110. 9.2.1
AQAP 2105
.4.9

Concessions.
DefStan 05-61
Pt1. para 6.1
AQAP 2105 para
4.8.4

Benefit to
Authority

DefStan 05-61 Pt1 Categorisation

para 7.4
AQAP 2105 para
4.8.4

The Supplier shall ensure

that only acceptable
products, intended for
delivery are released.

(d)

(e)
1-2

(f)
3-5

(g)
6-7

No Risk Plan No Risk Plan or Some evidence

or Register
Register
of Risk
Planning & Risk
Register

<89%

>90-95%

>95-<99%

All non-conforming product

offered to the Authority for
acceptance is accompanied
with a clear and
demonstrable benefit to the
Authority.

No benefits
demonstrated

a. All non-conformities
classified as Minor are to be
recorded and controlled in
accordance with the
contractor's QMS.

No evidence of Little evidence Some evidence

Records or
of Records or of Records &
Control
Control
Control

Page 11 of 14

Negligible
Minor evidence
benefit
of benefit
demonstrated

(h)

(i)
8-10

(j)
Remarks

Output of Management Review

> 95% of
meetings.
identified
Risks
Planned,
Managed &
Risk Register
Maintained

>99-100%

% of deliverables presented found

acceptable by the Customer. This
can include, design reports, reviews,
studies and prototypes as well as end
products.

100% Of
In terms of : TIME, COST,
Concessions PERFORMANCE.
have benefit
demonstrated.

>95%
Correctly
classified,
recorded &
controlled

(a)
AREA /
REFERENCE

(b)
QPI SPECIFIC

CRITERIA

b. Contractor's records of
Minor non-conformities will
be available upon request to
the Representative of the
Authority.
QPI SCORING - SUPPLIER ON CUSTOMER
Government
Furnished
Equipment.

(d)

(e)
1-2

(f)
3-5

(g)
6-7

(h)

(i)
8-10

Not accessible

Accessible
within 10
Working Days

Accessible
within 5
Working Days

>95%
Available

<89%

>90-95%

>95-<99%

>99-100%

(j)
Remarks

AQAP 2105 para

5.

Timeliness

GFA is delivered by DE&S

in accordance with the GFA
plan.

DefCon 611

Quality

GFA delivered by the DE&S

is fit for purpose.

Unsatisfactory Unsatisfactory

DE&S response to
applications for concessions
is within the agreed
timescale.

Response >30 Response >20 - Response >10 <20 Working

Working Days <30 Working
Days
Days

>95%
Measurement of the time it takes for
Response the DE&S to provide an answer to the
within 10
concession application.
Working Days

The DE&S quality

assurance resource
employed on the contract
are appropriately skilled

Have
Have
completed the completed the
Quality
Quality
Assurance
Assurance
Awareness
Practitioner 1
Course
Course

Licensed
Quality
Assurance
Practitioner
employed on
the contract

Response to
Concession
Applications.
AQAP 2105 para Timeliness
4.7.2

DE&S
Competence

Authority

Page 12 of 14

Did not fully

meet
requirements

Have
completed the
Quality
Assurance
Practitioner 2
Course

Percentage of GFA delivered on time

>95% meets Measurement of the adequacy of the

requirements GFA provided.

(a)
AREA /
REFERENCE
Communication.

(b)
QPI SPECIFIC

AQAP 2105 para

4.7.2

Contacts

AQAP 2105 para Customer

4.7.2
Perception

CRITERIA

(d)

(e)
1-2

(f)
3-5

(g)
6-7

The DE&S provides and

maintains points of contact
within its organisation.

No contact
details
provided

Not advised of
any contact
changes

Incomplete
contact list
provided

The DE&S provides

feedback to support
Management Reviews.

DE&S
Feedback
Poor

DE&S
Feedback
Adequate

DE&S Provides
Good
Feedback

(h)

(i)
8-10

(j)
Remarks

>95% Contact Complete list of DE&S personnel

list provided provided and updated as appropriate.
and
maintained
>95%
Feedback
provided

Measurement of the 'quality ' of the

feedback provided.

Information
requests

The DE&S response to

information requests is
provided within the agreed
timescale.

Provided
Provided within Provided within
within 40
30 Working
20 Working
Working Days
Days
Days

>95% of
responses
provided
within agreed
timescale.

Quality
Escapes

The DE&S feedback on

'quality' issues is provided
within the agreed timescale.

Feedback
Feedback
Feedback
within 40
within 30
within 20
Working Days Working Days Working Days
of occurrence. of occurrence. of occurrence.

Feedback Measurement of early notification of

provided
'quality issue' arising.
within agreed
timescale.

Page 13 of 14

Annex C to
Mandated QPI Guidance
Worked Example of a Contested Mandated QPI
Human Resource
1.

The Supplier determines that to fulfil the contract they will need 6
welders certificated to EN 1418 Approval of welding operators: Fusion and Resistance welding.

2.

They currently have 4 welders qualified to this standard and another

who is in the process of obtaining this qualification. They are currently
in the process of recruiting a qualified welder. It was not envisaged that
this would be a long term issue.

3.

The next review carried by the QPI stakeholders seeks to confirm that:
An additional qualified welder has been recruited.
The training of the unqualified welder has progressed or
completed.
The training records have been updated.

4.

The outcome of this joint review is that:

The recruitment of the qualified welder was unsuccessful
and an unqualified welder has been recruited. Training for
this person has been arranged for the next month.
The existing unqualified welders training has commenced
but was delayed due to sick leave. It will recommence in
two weeks time.

5.

The scoring for this QPI was, for each of the stakeholders, as follows:

Supplier Stakeholder QPI Achieved

Rational They had taken all possible steps to ensure that they had
sufficient resource to fulfil the contractual requirements
Customer Stakeholder Scored 6 out of 10
Rational The resource levels for completion of the contract were not
to the agreed level, out of a requirement of 6 the Supplier only had 4
qualified staff. There was concern that the shortage of qualified welders
would impact upon the contract.
6.

After some discussion a final score could not be agreed upon. The
issue was escalated to the nominated management representatives of
both Supplier and Customer. The outcome was that the agreed
contractual requirements had not been met.

Page 14 of 14