Fluke Calibration

Web Seminar Series

Understanding and Implementing
Decision Rules in the new
ISO/IEC 17025

Jeff C. Gust
Chief Corporate Metrologist
Understanding and Implementing Decision
Rules in the new ISO/IEC 17025:2017
• Your Presenter:
• Jeff Gust
• Chief Corporate Metrologist

• Your Host:
• Jovious Chua
Poll Question 1
Is your laboratory accredited to ISO/IEC
17025:2017?

A) Yes
B) Not yet
C) We are not Accredited to ISO/IEC 17025
Poll Question 2
Does your laboratory use decision rules to
make statements of conformity today?

A) Yes
B) No
C) Not Sure
Poll Question 3
Do you have customers or regulatory
organizations that specify a certain decision
rule?

A) Yes
B) No
C) Not Sure
Abstract

ISO 17025:2017 has an increased focus implementing decision rules

when deciding if an instrument is in or out of tolerance. This
presentation will provide a background on the fundamentals of
measurement decision risk and how decision rules are used to control
measurement decision risk. The presenter will also recommend some
decision rules that are relatively easy to implement and will provide
some of the statistical underpinnings of the decision rules and
communicates the probability of risk associated with the rules
When I ask for my instrument to be calibrated…

• What do I expect to get?

• A sticker?
• Low cost (preferably free)?
• Repair?
• An adequate procedure is used?
• (Accredited) certificate of calibration?
• What does calibration Mean?
• Adjustment to nominal?
• Measurement data?
• Uncertainty?
• Measurement Traceability?
• In or out of tolerance information?
• We need to use common definitions for this important term...
Language of Metrology

• Let’s all try to use the same language, the

International Vocabulary of Metrology (VIM)
• The VIM is available for free download at
http://www.bipm.org/en/publications/guides/
• For ease of understanding, some of the terms
used in this presentation are not always exact
definitions but are consistent with the VIM
definitions
Calibration Defined
• Calibration - operation that, under specified conditions, in a first step,
establishes a relation between the quantity values with measurement
uncertainties provided by measurement standards and
corresponding indications with associated measurement uncertainties
and, in a second step, uses this information to establish a relation for
obtaining a measurement result from an indication
• Jeff’s Translation – calibration is a measured value with an associated
uncertainty of measurement
• Calibration is not adjustment!
• Calibration is not “In Tolerance” or “Out of Tolerance”
• If you are performing calibration, you need to know about uncertainty
Statements of conformance
• Verification - provision of objective evidence that a given item fulfills
specified requirements
• Review the calibration data, compare to a specified requirement (i.e.
manufacturer specification) to determine In Tolerance or Out of
Tolerance condition
• If you need the measurements and uncertainty, ask for calibration
• If you need a statement of conformance (in/out of tolerance info), ask
for calibration and verification
• Maybe best call this “Calibration Service”
ISO/IEC 17025:2005 on Conformance

• (reporting the results) 5.10.4.2 When statements of compliance are

made, the uncertainty of measurement shall be taken into account

• Changed from using the term compliance to conformance

• The term compliance is used for legal requirements, conformance is
used for statement of suitability
ISO/IEC 17025:2017

• (Review of requests, tenders and contracts) 7.1.3 When a customer

requests a statement of conformity to a specification or standard for
the test or calibration (e.g. pass/fail, in-tolerance/out-of-tolerance) the
specification or standard, and the decision rule shall be clearly
defined. Unless inherent in the requested specification or standard,
the decision rule selected shall be communicated to, and agreed
with, the customer
ISO/IEC 17025:2017

• (Reporting of results) 7.8.6.1 When a statement of conformity to a

specification or standard is provided, the laboratory shall document
the decision rule employed, taking into account the level of risk (such
as false accept and false reject and statistical assumptions)
associated with the decision rule employed and apply the decision
rule.
ISO/IEC 17025:2017

• 7.8.6.2 The laboratory shall report on the statement of conformity such

that the statement clearly identifies:
a) to which results the statement of conformity applies;
b) which specifications, standard or parts thereof are met or not met;
c) the decision rule applied (unless it is inherent in the requested
specification or standard).
• NOTE For further information see ISO/IEC Guide 98-4.
• Statements of conformity are in addition to reporting measurement data
and uncertainty of measurement
What’s a decision rule?

• (Terms and Definitions) 3.7 – Decision rule: rule that describes how
measurement uncertainty is accounted for when stating conformity
with a specified requirement
Why are decision rules important?

• When performing a calibration and subsequently making

a statement of conformance with an identified
metrological specification (such as in or out of tolerance
to manufacturer’s specifications) there are two possible
outcomes
• You are right
• You are wrong
• Each measurement has an associated uncertainty, and it
is the uncertainty that can affect your chance of being
right or wrong
Measurement Decision Risk

• If your uncertainty is large compared to the specified requirement, how

confident are you in your declaration of In Tolerance or Out of
Tolerance?
Measurement Decisions

• When an instrument is declared in tolerance but the

instrument is actually out of tolerance, it is referred to as
a false accept (FA)
• When an instrument is declared out of tolerance, but the
instrument is actually in tolerance, it is referred to as a
false reject (FR)
• False rejects cost money by requiring unnecessary
adjustment, repair or replacement
• False accepts cost everybody money by putting out of
tolerance test equipment in the field
• Insert horror story here….
Measurement Decisions Involve Risk

• The chance of making a false accept or false reject can

be understood as a probability, which is usually
expressed as a percentage
• When a customer requests a calibration they
• Usually don’t worry about false reject risk
• May specify a maximum acceptable false accept risk
• How much false accept risk is too much?
• 1%?
• 5%?
• 10%?
• 33%?
Controlling risk

• Measurement decision risk can be controlled through the application

of Decision Rules
• Two types of Decision Rules
• Rules that control risk by specifying a minimum uncertainty (3:1 or
4:1 rule)
• Rules that controls (FA) risk by testing to a value that is smaller than
the specification (guardbanding)
N:1 rules

• Evaluate the specification you are calibrating to

• Evaluate the uncertainty associated with the measurement
(calibration)
• Determine the Test Uncertainty Ratio

4:1 TUR ≈ 0.9 % FA, 1.6 % FR*

3:1 TUR ≈ 1.1% FA, 2.3% FR
2:1 TUR ≈ 1.5% FA, 4.3% FR
1:1 TUR ≈ 2.2% FA, 13.4% FR

*Evaluated using joint probability, 95% End Of Period Reliability

Guardbanding

• Decreases False accept risk by developing a “test limit” that is less

than the specification limit

False
Accept
Excess risk

Risk
Acceptable risk

Test
Limit

Specification
Limit
Guardbanding

STD
Indeterminate
Region

Indeterminate
Region
UUT

-SL -TL Nom Meas +TL +SL

Guardband Selection Methods

• For a 2:1 TUR

• Root difference of Squares TL = SL − U
2 2
95%
• ≈ 0.6 % FA, 8.2 % FR
• ISO 14253-1 (U95)
• ≈ 0.1 % FA, 35 % FR TL = SL −U 95%
• Dobbert method
• ≈ 2 % FA, 10 % FR
TL = SL −U 95%×M
• Zero Guardband
• ≈ 1.5 % FA, 4.3 % FR TL = SL
• All evaluated with joint probability model
• ISO 14253-1 evaluated by conditional probability 2.5% FA
A Real World Example
• Measuring the output of a Fluke 5500A at 10 V DC using a Fluke 8846A
• Fluke 5500A 1 year specification = 50 ppm of output plus 50 µv = 550
µv (99% level of confidence, normal dist.)
• Fluke 8846A 1 year specification = 0.0024% of reading plus 0.0005% of
range = 290 µv (99% level of confidence, normal dist.) = 225 µv @ k=2
• Completing a full uncertainty analysis, assuming that resolution,
repeatability, and calibration uncertainty for 8846A insignificant = 225 µv
• NOTE: PROPER ESTIMATION OF MEASUREMENT UNCERTAINTY
IS CRITICAL
• TUR = 2.4:1
A Real World Example

• Is a 2.4:1 TUR acceptable?

• Depends on the customer
• Time to make a measurement
A real world example
A real world example

• (Note: 8846A is in tolerance to its specifications)

• Manufacturer’s specification for Unit Under Test equals +/- 550 µv
• DMM reads 410 µv above nominal
• The TUR is 2.4:1
• For 95% UUT in tolerance and 95% level of confidence, the false
accept probability is 1.3%
• Is this measurement acceptable?
100% Guardband
ISO 14253-1,
• Guardband is manufacturers specification minus the measurement
uncertainty
• Original specification 9.999450 to 10.000550 volts
• Uncertainty is 225 µv
• For this case, the Guardband limits would be set to 9.999675 to
10.000325 volts
• Any measurement within this limit is in tolerance
• Any measurement between 9.999450 to 9.999675 and 10.000325 to
10.000550 volts is indeterminate (60% of the original specification
• Was our measurement of 10.000410 volts acceptable using this
method? What do we report to the customer?
100% Guardband
ISO 14253-1
• Reporting an indeterminate result is not what most customers want to
hear.
• Conditional probability manages false accept to almost zero, but false
reject is unacceptably large, and creates the undesirable
“indeterminate zone”
• False rejects cost customers money by performing unnecessary out
of tolerance investigations and possible recall of products.
Joint Probability
RDS Guard band method
• Joint Probability considers both UUT and standard
• On method is the Root Difference of Squares

TL = SL − U 2 2
Std
• For same scenario as discussed
• Test limits (Guardband) are 9.999556 to 10.000443 volts
• False accept risk is reduced to less than 1%, similar to what occurs
with 4:1 TUR. This increases the false reject probability to about 3%
• Using our example can we now state with confidence that our unit is
in tolerance?
8846A measures 5500A
What now?

• The measurement is within the Guardband limits

• Is the unit really in tolerance?
• Accept RDS as acceptable risk and move on
• What if the measurement is greater than
Guardband limit but less than tolerance for this
result?
• Could we improve our measurement uncertainty
(and TUR) and try again?
One last measurement

• Instead of using the 8846A, try using an 8508A

• 1 year specifications 3.5 ppm of rdg + 0.2 ppm of range = 39 µv (95%
confidence level, 1 yr +/- 5C)
• TUR = 14:1
• The higher TUR reduces the probability of false accept and false
reject significantly
• Let’s make a measurement
• What is the result?
8508A measures 5500A
One last measurement

• Fluke 5500A is measured as 10.000629 volts

• Recall that the 5500A tolerance was 550 µv
• Better equipment with a smaller uncertainty demonstrates that
the 5500A is actually out of tolerance!
• Recall that the 8846A with the RDS method had approximately
1% risk of false acceptance
• For the purpose of this demonstration, the 5500A was adjusted
out of tolerance high and the 8846A was adjusted to be in
tolerance on the low side, in order to demonstrate that the 1%
risk is real and does happen 1 out of 100 times
• Similar to playing roulette, betting equal on black and red, and
the ball landing on the zero or double zero turquoise
• On the grand scheme, 1% risk is still very acceptable to use
Jeff’s philosophies on decision rules
• People don’t pay us a lot of money to receive an indeterminate
result
• Although the true value can exist anywhere within the uncertainty
bars, the best estimate of the true value is the measurement itself
• If a measured value is within the tolerance limits, it is highly likely
that it is in tolerance
• If the measurement is between the guardband and the tolerance limit, better
to use conditional pass over indeterminate
• If we can have sufficiently small risk without using a guardband,
this is the best approach
• If we have to guardband, use a method:
• That optimizes false accept and false reject probabilities
• Is easy to apply
Conclusions

• The customer and the calibration lab need to communicate in order

to understand what is provided by the calibration service
• Clarify if a statement of conformance is needed or not
• Agree on a decision rule and communicate the level of risk
associated with it
• Report the information on the certificate of calibration
• Lots of NCSLI conference papers available on the subject
• NCSLI 173 (Metrology Practices) committee has lots of experts that
can help
• New ILAC G8, Guidelines on Decision Rules and Statements of
Conformity is very helpful on this subject, free at (www.ilac.org)
• Risk is still risk, but you need to live with an acceptable amount
For Further Reading

• (BIPM) JCGM 106:2012 Evaluation of measurement data – The role of

measurement uncertainty in conformity assessment
• Deaver, Dave “How to Maintain Your Confidence (in a World of Declining Test
Uncertainty Ratios)
• Deaver, Dave “Guardbanding with Confidence”
• Deaver, Dave “Guardbanding and the World of ISO Guide 25 Is There Only
One Way?”
• Dobbert, Michael “A Guard Band Strategy for Managing False Accept Risk”
• NCSLI RP-18 Estimation and Evaluation of Measurement Decision Risk
• ILAC G8:09/2009 Guidelines on Decision Rules and Statements of
Conformity
Thank You for Attending!

Jeff Gust
Chief Corporate Metrologist
Fluke Corporation
6920 Seaway Blvd
M/S 275G
Everett, WA 98203
Direct: +1 425 446 5471
Email: [email protected]
Web: www.fluke.com www.flukecal.com
Twitter: @Jeff_Gust