CHEM 4430

Practices in Testing Laboratory

Proficiency tests
(for external QC)

Reference: R.E. Lawn, M. Thompson and R.F. Walker, "Proficiency Testing in

Analytical Chemistry", The Royal Society of Chemistry, 1997

1
Introduction:

What is Proficiency Testing?

• A proficiency testing scheme comprises the regular distribution of
test materials to participating laboratories for independent testing.
testing

• The results are returned to the organizer of the scheme who makes
an analysis of the results and report them to all of the participants.

• The primary function of the scheme is to assist the participants

to detect shortcomings in their execution of the test procedures
and apply suitable remedial measured to make up any deficiency.

2
Obj ti
Objectives off Proficiency
P fi i T ti
Testing:
(a) The provision of a regular, objective and independent assessment of the
accuracy of an analytical laboratory
laboratory’ss results on routine test samples.
samples

(b) The promotion of improvements in the quality (accuracy) of routine

analytical
y c ddata..

Other objectives:
• Provision of support to laboratory accreditation activities
• The identification of competent laboratories for regulatory, commercial or
other purposes
• An assessment of the data quality of a particular test sector, rather than of
individual laboratories
• A comparison of the performance of different analytical methods used by
participant laboratories for a particular measurement
• Th production
The d ti off materials
t i l suitable
it bl for
f training
t i i purposes or quality
lit control
t l

3
Interlaboratory Studies of Different Purposes

4
The statistical distribution of participants’ results
In data collected under repeatability
p y conditions,, there is an assumption
p that
the frequency distribution of the results will be close a normal distribution
due to numerous, small, independent errors made at the many stages of the
manipulations
p in the analytical
y procedure.
p

ncy
Frequen

No. of
laboratories
with results at
x

5
In proficiency test,
test data are not produced under repeatability conditions and
often not even under reproducibility conditions, because several analytical
methods or many variants of the same method may be used by the different
participants in a round of the test.
test

outliners

Q
Quasi-normal distribution off results from
f a round off a proficiency
p f y test for
f the
determination of glucose in blood serum
6
Characteristics of the distribution
• Bell-shape distribution or quasi-normal distribution

• Heavy tails which means that high proportion of observations are far
from the median

• With some outliners which are the results that are far from the central
tendencyy of the distribution and theyy are most pplausiblyy ppart of a
different distribution

• Only the central part resemble the normal distribution

7
Oth di
Other distribution
t ib ti
Example of a bimodal distribution

Measurements of alkaline phosphatase in blood serum from a round of a

proficiency test, showing the bimodal distribution resulting from the use of two
diff
different
t methods
th d among the
th participants.
ti i t

8
Consensus and Assigned Value

If the assigned value is the

true value, the results
show a negative bias in
the method.

Assigned value: the value set by the organizer

Consensus: a broad agreement between laboratories, e.g. mean, median and
mode
d

9
Histogram of results from a round of
proficiency test for determining lead
in a rock

Consensus: mode

IIn other
th cases: may use median,
di
mean or other statistical techniques

10
Approaches to performance scoring
“q-score” Result
reported by a
participant

( x − xa ) Q-score of the analysis

q= of silica

xa Assigned
g value
of the test
material

In the same round

The dispersion of q would tend to

vary between
b analytes
l andd matrices
i Q-score of the analysis of
and to a degree, for different holmium
concentrations of the same analyte

11
“z-score”
( x − xa )
z= The parameter σ is known as the target value

σ
for standard deviation

A judicious choice of the value σ would result in a set of scores that would
be dimensionless, zero centred and bounded approximately by a similar
range

z-scores

12
Classification of participants
• Z-score are best to alert participants to an unexpected source of error in
the
h analytical
l i l system. A value l off |z|≥2
| | 2 acts as a warning
i off potential
i l
problems and a value of |z|≥3 is an indicator that remedial action must be
undertaken.

• A classification may base on

|z|≤2, satisfactory
2<|z|<3, questionable
|z|≥3 unsatisfactory
|z|≥3,

• Some laboratories use the classification for licensing, accreditation or

commercial purposes,
purposes e.g.
e g 4 out of 5 in past 5 rounds of a proficiency test
with |z|<2.

13
Examples of z-score plots

Plot of z-scores obtained by Plot of z-scores from successive

diff
different analyses
l in
i a roundd off a rounds of a proficiency test
proficiency test scheme. scheme obtained by a
participant for a single analyte

14
Example of measurements (fat in a meat product) from a round of a proficiency
test where the observed dispersion is greater than that specified by the target
value for standard deviation. The vertical lines show the area bounded by –
2<z<2, i.e.
i theh satisfactory
if result.
l

15
Measurements of fat in a meat pproduct in a round of pproficiency y test. The
apparently wide dispersion is explained when the results are distinguished by
method. White bars show acid hydrolysis method and hatched bars other
methods.

16
Combination of scores
The combination of several z-scores to produce an overall statistic,
e.g. summaries the performance of a laboratory in a particular test
over a period of a year.
year

The rescaled sum of zz-scores

scores (RSZ)

zi
RSZ = ∑
i n

17
Z-score data for a single participant demonstrating that a combined score
such as RSZ can often hide the fact that the participant is consistently not
performing satisfactorily for a particular analyte

18
The sum of squared z-scores
z scores (SSZ)

SSZ = ∑ z 2
i
i

Z-score Z-score Z-score Z-score Z-score Z-score RSZ SSZ

08
0.8 -00.9
9 -11.2
2 18
1.8 -00.2
2 14
1.4 0 69
0.69 81
8.1
0.8 -0.9 -1.2 1.8 -3.9 1.4 -0.82 23.2
0.8 5.0 -1.2 1.8 -3.9 1.4 1.59 47.5

- SSZ avoids the cancellation of large scores of opposite sign.

- Usingg Chi-Square
q distribution to check for the satisfactoryy of the
SSZ result.

19
Ranking of participants
There is a strong tendency among participants (and to a lesser degree among
proficiency test organizers) to want a list of participants that is ranked on the
basis of the z-scores pproduced within the round.

• Widely perceived to represent the 6

relative ability of the participants and
4

z- score
to encourage low ranking participants
to better performance 2

• As advertisement and promotion 0

1
34
21
7
39
16
29
24
36
14
26
37
12
23
15
4
19
20
11
5
40
33
17
32
22
25
13
9
30
8
31
2
35
10
3
28
38
6
18
27
Laboratory Number

What could be misleading?

g

20
Applicability of Proficiency test
• As a quality control procedure where the laboratory has a large
throughput of routine samples and tests of the type covered by the
scheme

• Staff training

• Maintenance of expertise

• Demonstration of commitment to quality

• For sectors where reference materials are difficult or impossible to

obtain, the materials distributed in a proficiency test scheme
provide a substitute for CRMs.

21
Long term review of proficiency testing
Efficacy measure Ema
The percentage of participant laboratories producing results of acceptable
quality
Shows improvement after the first round

Efficacy measure for laboratory

uundertaking
de ta g asbestos fiber
be
counting

22
Efficacy measure for laboratory undertaking measurements of toluene

In general, the efficacy measure tends to improve over the first few rounds
and then stabilize at some “plateau” value.

23
Efficacy measure for Efficacy measure for
laboratory undertaking laboratory undertaking the
analysis
y of a wide rangeg of determination of cadmium in
analytes blood

24
Limitations of Proficiency testing

The retrospective nature of proficiency testing

The organization structure of a proficiency testing scheme – the return
of results to an external coordinator, the subsequent processing of
results and the distribution of a performance report to the participant –
entails an unavoidable time delay in the assessment of the laboratory
data quality. The performance scores reflect the quality of a
laboratory’ss analyses at some fixed point in the past. It is not
laboratory
acceptable to use proficiency testing to ensure day-to-day quality of a
laboratory’s data.

25
Characteristics of the p
proficiency
y test materials
• The proficiency test material should resemble the routine test
materials closelyy and the assigned
g values for the various analytes
y must
be reliably established.

• g y of the test materials must be well verified.

The homogeneity

• If the assigned values for the material have to be established from the
consensus of the participant results, it should be realized that such
assigned values could be biased. Consequently, performance scores
based on such a value could also be in error.

26
The analyses covered by a proficiency testing scheme
• In laboratories that undertake a wide range of analyses, only a small
fraction of them can be subjected to proficiency test. This fraction
has to regarded in some sense as representative of the general
performance
f off the
h laboratory.
l b Th
There i no guarantee that
is h this
hi is
i a
correct assumption.

• The performance
Th f d
data provided
id d by
b proficiency
fi i tests are often
f
restricted to a small range of particular sample types. It is not
known for closely related tests to be conducted with different
d
degrees off accuracy among a group off laboratories
l b t i

• Proficiency testing in specific tests should result in a general

improvement of the internal quality control system,
system and that in turn
should result in improvements in the quality of all the analytical
procedures. However, there is no experimental evidence to support
this
this.

27
z-scores for Zn and Ca determined by the same participants on the
same material,
material both by flame atomic absorption spectroscopy.
spectroscopy

28
The disclosed/declared nature of proficiency testing
• Extra effort may be applied to the proficiency test samples in the
l b t i when
laboratories h comparedd with ith that
th t normally
ll applied
li d to
t routine
ti
samples, e.g. analyzed by the most experience person in the
laboratory, the material may be analyzed in duplicate where single
analyses are the norm in routine work.
work Thus,
Thus the test may not reflect
the normal performance of the laboratory.

• The laboratory may assign an internal auditor to examine the records

to check whether proficiency test samples are being afforded special
treatment.

29