RUNNING YOUR OWN PROFICIENCY TEST

R.R. Cook.

Metrology Training International, 7 Dallas Ave., Hughesdale, VIC., 3166.

Abstract.

All calibration laboratories and those testing laboratories that do some calibration and are accredited by
NATA to ISO 1725 are required to participate in at least one proficiency test prior to being accredited.
Once accredited the laboratory is required to participate in a proficiency test at least once every year
unless they are accredited for only one or two measurement groups in which case they must participate
at least every two years. Proficiency tests may also be required when significant extensions or
variations of the scope are requested. There are a number of proficiency testing providers, but some
work is very specialised and not covered by these organisations so the laboratory needs to organise its
own tests. This paper describes how a small laboratory can run an effective proficiency test and draws
attention to some potential pitfalls.

Keywords: proficiency testing, ISO 17025 accreditation requirements, laboratory intercomparison.

1 Introduction volume of the work. It then falls to the

laboratory to organise and undertake its
NATA documents sent to applicant own Proficiency Testing (PT). The
laboratories now state that: elements of the process along with
advice on potential problems are
“Each applicant or accredited facility is described in this paper.
required to participate in appropriate
proficiency testing activities. A measurement audit may be
Measurement Audits are considered a considered a simplified version of a
form of proficiency testing. proficiency test that involves only the
laboratory being appraised using
Participation in proficiency testing or previously qualified artefacts or
measurement audits may be required, samples.
as follows:
2 Why bother?
a) prior to gaining accreditation;
b) when requesting significant There are a number of advantages,
extensions or variations to the scope of over and above NATA requirements, to
accreditation; participating in a PT. Proficiency tests:
c) as identified by the accreditation
• give an independent review of
body.
results
• provide comparisons of
Facilities are encouraged to participate performance relative to peers
in as broad a range of proficiency • satisfy accreditation requirements
testing activities as practicable and • educate staff through learning of
available, but ideally at least once every other methods
two years for each major area of • verify methods and procedures
calibration, measurement or test.” • satisfy regulators
• identify problems not otherwise
See reference 1 for general policy and detected
others such as reference 2 for more • enhance accuracy by providing
specific testing field policy. targets (in relation to peers).
• monitor analysis performance
There are a number of experienced • assess laboratory performance
Proficiency Testing Providers but they • build confidence in the laboratory
are unable to cover every calibration clients and amongst the laboratory’s
activity undertaken in Australia due to staff
the specialisation nature or limited • demonstrates competence to all

Metrology Society of Australia Conference 2009 PREPRINT Page 1

 3 The first steps instruction to measure the output. It is
not recommended that anything so
The laboratory needs to identify the open or undisclosed be presented for a
particular test or calibration to be the PT. A PT should not (normally) require
subject of the PT. There must be at solving a puzzle.
least two participants, the reference
laboratory and the participation Laboratories considering undertaking a
laboratory and there must be agreement PT should refer to reference 3, ISO
by all on the nature of the PT. In some GUIDE 43-1 Proficiency Testing by
cases both laboratories will be of similar Interlaboratory Comparisons - Part 1:
capability so the concept of a higher Development and Operation of
capability reference laboratory providing Laboratory Proficiency Testing
the reference value needs revision. For Schemes for more details.
some types of PT a consensus value or
the median value will be a more 4 Action Plan
appropriate reference.
Once the type of test, artefact or sample
The laboratory (or organisation) details have been agreed and sources
responsible for running the PT must be of these identified the plan for the PT
identified and agreed by all program can be completed. This will
stakeholders. This laboratory will cover:
manage the process including the  Schedule for obtaining funding
analysis and final project report. and purchase of the artefact or
samples if required.
The more complete the description of  Schedule for obtaining the
the work the easier the program will be artefact or samples if required.
to run and analyse.  Setting of the dates on which
the test item is to be sent, to
The next step is to obtain a suitable whom and in what sort of
artefact or sample. Consider an pattern.
example where the intention of the PT is  Dates for reporting results
to demonstrate competence in  Date of issue of report of the
measuring a standard capacitor in the PT
vicinity of 100 pF at 1,000 Hz at 23 C. Arrangements for testing by a
reference laboratory may also be
The capacitor needs to have high needed.
reproducibility and stability. It should
not be exactly 100.000 pF and may 5 Circles, Stars or Spokes?
have a significant temperature
coefficient, but it must have stable and For two laboratories the distribution plan
reproducible values at a stable is simple but if there are more
temperature over the time the PT will be laboratories a decision must be made
in progress. as to whether the artefact/sample needs
to be returned to the reference lab after
A material sample needs also to be each lab has done its test (Star pattern)
stable and drawn from a homogenous or only at the start and end (Circle) or
stock. some intermediate arrangement (Wheel
The reasoning for “off nominal” values and Spoke pattern). The stability of the
is that unless the item under test has a artefact and duration of the PT will
measurable offset from nominal the determine which is best. PT with
ability of the laboratory is not so clearly samples would follow the star pattern.
exercised as if the results are the same See Fig 1.
as the nominal value. Indeed if it were
known that the artefact was very stable The geographic separation of
and adjusted to nominal then a lazy laboratories may also affect which
laboratory could perform a perfunctory model or variant is used.
test and just report the nominal value
with their claimed uncertainty. 6 Getting it there in good order
Proficiency testing should require the
laboratory to make their best effort. The Two issues need to be carefully planned
US Navy used to run “tricky” PTs where and managed. First the packing and
there were several oscillators connected transport of the artefact or sample
inside a sealed box with the simple needs care. The package must be

Metrology Society of Australia Conference 2009 PREPRINT Page 2

 robust and protect the contents from
rain, bumps, vibration, low or high 8 After the testing is over
temperature and an occasional drop if
the artefact or sample cannot withstand Once the testing is complete and all
this by itself. A reliable carrier must be reports received the controlling
identified, one who will not leave the laboratory can analyse and report the
package on the tarmac for 24 hours or results. A copy of the report should also
allow staff to throw and perhaps drop or be forwarded to NATA as evidence of
run over the package. the fulfilment of this condition of
accreditation.
Some samples have short life times or
change significantly over time and must Even when NATA has not requested a
be delivered within a small window of PT, any time a laboratory partakes in
time. The commitment of the carrier to one, notifying NATA of the test and its
this end is crucial and must be agreed outcomes demonstrates a proactive
in writing. The recipient laboratory must approach to maintaining the highest
be given the details of the despatch standards of testing and calibration.
immediately the item has been collected
from the provider by the carrier and the 9 Analysis.
recipient laboratory must acknowledge
receipt and the condition of the artefact Reference 4, ISO 13528:2005 Statistical
as received as soon as practicable after methods for use in proficiency testing by
the package arrives. This ensures that interlaboratory comparisons covers this
if there is a problem with the transport topic.
and delivery that it is detected promptly,
giving the best chance for recovery and There is a general similarity in the
continuance of the program. analysis methods for calibration and
testing, but the most appropriate
6 Knowing what to do method depends on the nature of the
test or calibration. A truncated version
The instructions to the participating of the recommended procedures
laboratories must be both follows.
comprehensive and clear. Everything
that the laboratory is required to test 9 By the numbers
and report must be stated. Test
conditions and allowable variations from For artefact calibrations the usual
standard methods must be stated. It is method of analysis involves calculation
a wasted exercise if the laboratory uses of the En number and graphically
its standard method but omits one or summarising the results. The En
more requested test parameters. The number is calculated by using the
allowable retention time for the artefact, following equation.
the due date for the report and
instructions about packing and on
forwarding the artefact must have been LAB VALUE  REFERENCE VALUE
given in a written document prior to the En 
PT commencing so arrangements can
2
U LAB  U REF
2

be made for an expeditious response

when the time comes. where:

The participating laboratories must En is the normalised Error and

commit in writing to achieving the
schedule. LAB VALUE is the participating
laboratory’s result
7 Getting it done
REFERENCE VALUE is the Reference
Providing there has been effective Laboratory’s result
communication on what is required the
testing and or calibrating is the simplest ULAB is the participating laboratory’s
part of the process. After all this is reported (expanded) uncertainty
supposed to be an activity the
laboratory is proficient at. UREF is the reference laboratory’s
reported (expanded) uncertainty

Metrology Society of Australia Conference 2009 PREPRINT Page 3

 In general if |En| ≤1 the results are See Fig 2 which shows the reported
considered as satisfactory and if values for a number of laboratories plus
|En| >1 the results are considered as the reference laboratories.
unsatisfactory. Generally, the desired
outcome is for the value to be as close Some drift in the artefact is present so
to zero as possible. the value for the trend line would be
used as the reference value.
For analytical chemistry and many other
testing fields the above method is For other types of testing the
modified. laboratories may be plotted in rank
order. In Fig 3 the z scores are plotted
Instead of an En number, the z score is with the largest negative value first and
calculated. the largest positive value last.

The equation for this is: A variation on sending one artefact or

one sample is to send two or sets of
LAB RESULT  MEAN  RESULT pairs. For calibration laboratories the
z artefacts can have similar values but
STANDARD DEVIATION
more can be learned about systematic
errors if the artefacts are different in
The standard deviation and mean are value by a factor of 10 or more. The
calculated from all the results. differences would be converted to
proportional parts and a Youdon plot
If |z| ≤2 the results are considered as constructed. The point on the Youden
satisfactory and if plot is the coincident point of the
|z| >2 and <3 the results are projection of the results for sample A (x
considered to be a warning sign and if axis) and sample B (y axis). See Fig 4.
|z| >3 the results are considered as
unsatisfactory and action is required. The results should ideally all lie within a
circle or ellipse. Any tendency for the
For some tests, especially when the results to lie in a line inclined at forty-
number of participants is small and a five degrees indicates a common
robust analysis is required then the systematic error.
median may be used instead of the
mean, and the normalised Similarly the results of other tests such
interquartile range (IQR x 0.7413) may as chemical analysis can be plotted for
be used instead of the standard pairs of results on different samples and
deviation. an ellipse equal to two or three times
the standard deviation (or normalised
Other statistical scores that may be IQR) constructed. Results falling
used include the ζ score, which is outside these limits are outliers. See
similar to the En score but uses Fig 4.
standard uncertainties instead of
expanded uncertainties, the z’ score 11 Follow up action
and the Ez score. See ISO 13528 for
details. Any laboratory returning a high En
number or high z score needs to
10 Seeing is instructive investigate the reasons. Similarly any
laboratory with a result otherwise
The results of the analysis need to be identified as an outlier needs to identify
presented in the PT report. Graphical the cause and take corrective action.
displays of the test results may be more
meaningful and easier to follow than A high En may arise from an
tables of the numeric analysis results. unrealistically low estimate of the
For calibrations the results and their uncertainty achieved.
associated uncertainties may be plotted
on a time line with a line joining the A high z number is likely to be due to an
reference laboratory results so the error, such as a transcription error,
participants can not only see their misreading an instrument,
results in comparison to others and the instrumentation problem or a poor
reference but can also see if drift was a technique. It could also arise from an
significant factor. unsatisfactory test method.

Metrology Society of Australia Conference 2009 PREPRINT Page 4

 Similar reasons could account for outlier
results in a Youden plot. Consistently
low results indicate a systematic error
but results that fall in different positions
outside the acceptance ellipse indicate
poor measurement control, arising from
instruments, methods or execution of
the methods.

12 Last words

Experience shows that frequently the

inaugural PT test in any field is likely to
show unexpected disagreement among
laboratories. Results improve following
investigation and remedial action by the
laboratories.

Sometimes there is a requirement for

confidentiality of the laboratories
identities and their results are identified
only by a number. This is becoming
less common as transparency is
required by many regulators and the
laboratories want their good work to be
visible to all.

The cost of programs is not

insignificant, however most laboratories
can readily see the value in
participation. Apart from the benefits
listed in the introduction, better test
methods and increased staff morale
lead to reduced costs as well as better
results.

References
1. Policy Circular 2 – March 2009,
Proficiency testing policy.
NATA
2. Technical Circular 8 –
November 2007. Assuring the
quality of tests and proficiency
testing policy for the field of
Construction Materials Testing.
3. ISO/IEC Guide 43-1:1997
Proficiency Testing by
interlaboratory comparisons –
Part 1. Development and
operation of proficiency testing
schemes.
4. ISO 13528: 1995 Statistical
methods for use in proficiency
testing by interlaboratory
comparisons.

Metrology Society of Australia Conference 2009 PREPRINT Page 5

 Fig 1. Illustration of samples sent in a star pattern and an artefact sent in a circle pattern (solid
arrows). The dotted arrows show the laboratories reports being sent to the organising
laboratory.

Participating Laboratories
5

4
1
3

2 Uncertainty

Reference Laboratory
Fig 2. Illustration of plot of test results with uncertainties of measurement. Note the reference line
joining the two reference values. This compensates for drift in the artefact. The x axis is time and the y
axis is the reported value (scale usually offset for clarity)

Metrology Society of Australia Conference 2009 PREPRINT Page 6

 Fig 3. Graphical representation of results showing z scores ranked in progression from most negative to
most positive. From PT Australia.

95%
probability
Sample 2

Youden
Plot

9
13
Outliers

Sample 1
Fig 4 Youden plot for two samples. Laboratories 9 and 13 are outlier laboratories and need to investigate
the cause of these results.

Metrology Society of Australia Conference 2009 PREPRINT Page 7