Introduction to the Risk-Based Approach

Charlie Wakeham
ISPE Philippines Conference April 2015

©2015 Waters Corporation 1

 The Risk-Based Approach

©2015 Waters Corporation 2

 Why use a Risk-Based Approach?

 FDA Final Report –

Executive Summary:
… Part 11 Guidance
With the issuance in 2003 of the
guidance for industry Part 11,
Electronic Records, Electronic
Signatures — Scope and Application,
many barriers to scientific and
technological advances were removed,
and the use of risk-based approaches
to managing computer systems is
encouraged.

©2015 Waters Corporation 3

 Why use a Risk-Based Approach?

 GAMP 5 Main Body

2.1.4 Science Based Quality Risk Management
− Quality Risk Management is a systematic
process for the assessment, control,
communication, and review of risks.
− Application of Quality Risk Management
enables effort to be focused on critical
aspects of a computerized system in a
controlled and justified manner.
− Controls are developed to reduce risks to an
acceptable level. Implemented controls are
monitored during operation to ensure
ongoing effectiveness

©2015 Waters Corporation 4

 Why use a Risk-Based Approach?

 GAMP Testing GPG 2nd Edition

2.4 Science Based Quality Risk
Management
…The controls identified by the quality
risk management process should form
the basis for designing tests that verify
and challenge the functions within the
system; with each control being
considered as a starting point for
generating test cases. This allows critical
aspects of a computerized system to be
identified and tested adequately.
 Tracing that all controls have been
verified as effective is part of effective
test reporting and determination of
residual risk.

©2015 Waters Corporation 5

 Regulations on Risk – Annex 11

 Annex 11 is heavily risk-based:

– 1. Risk Management
Risk management should be applied throughout the lifecycle of the
computerised system taking into account patient safety, data
integrity and product quality. As part of a risk management system,
decisions on the extent of validation and data integrity controls
should be based on a justified and documented risk assessment of
the computerised system.

– 3. Suppliers and Service Providers

3.2 The competence and reliability of a supplier are key factors when
selecting a product or service provider. The need for an audit should
be based on a risk assessment.

©2015 Waters Corporation 6

 Regulations on Risk – Annex 11

– 4. Validation
4.1 The validation documentation and reports should cover the
relevant steps of the life cycle. Manufacturers should be able to
justify their standards, protocols, acceptance criteria, procedures and
records based on their risk assessment.

4.4 User Requirements Specifications should describe the required

functions of the computerised system and be based on documented
risk assessment and GMP impact. User requirements should be
traceable throughout the life-cycle.

©2015 Waters Corporation 7

 Regulations on Risk – Annex 11

– 6. Accuracy Checks
For critical data entered manually, there should be an additional
check on the accuracy of the data. This check may be done by a
second operator or by validated electronic means. The criticality and
the potential consequences of erroneous or incorrectly entered data
to a system should be covered by risk management.

– 16. Business Continuity

For the availability of computerised systems supporting critical
processes, provisions should be made to ensure continuity of support
for those processes in the event of a system breakdown (e.g. a
manual or alternative system). The time required to bring the
alternative arrangements into use should be based on risk and
appropriate for a particular system and the business process it
supports. These arrangements should be adequately documented
and tested.

©2015 Waters Corporation 8

 PIC/S PI 011-3

PI 011-3 is the ‘how to inspect’ guide for PIC/S inspectors

©2015 Waters Corporation 9

 PI 011-3

4.5 When a GxP inspector has to assess an installed computerised

system at a regulated user’s site, s/he may consider some, or all, of
the elements shown in Figure 1: “Computerised system”, (viz.: the
controlling system and the controlled process in an operating
environment). The inspector will consider the potential risks, from the
automated system to product/material quality or data integrity, as
identified and documented by the regulated user, in order to assess
the fitness for purpose of the particular system(s). The company’s risk
assessment records may also be referred to as part of this process.
The inspector’s assessment may also involve a consideration of system
life cycle, quality assurance measures, validation and operational
control evidence for the controlling system, as well as validation and
operational experience with the controlled process.

©2015 Waters Corporation 10

 PI 011-3 Checklists

 1. Identify system
 2. Produce URS
 3. Determine validation strategy
• Risk Assessment
An initial Risk Assessment should be carried out during validation
planning. Further assessments should be performed as specifications
are developed.
• Assessment of system components
System components should be assessed and categorized to determine
the validation approach required. The output from this assessment
will feed into the Validation Plan.
• Supplier assessment
Suppliers should be formally assessed as part of the process of
selecting a supplier and planning for validation. The decision whether
to perform a Supplier Audit should be documented and based on a
Risk Assessment and categorization of the system components…
©2015 Waters Corporation 11
 The recommended
approach to validation
is to test all functions
in the system equally

©2015 Waters Corporation 12

 Risk-Based Approach

Data
Integrity

Product
Quality

Patient
Safety

©2015 Waters Corporation 13

 Risk Integration into the Life Cycle

©2015 Waters Corporation 14

 Focus of Effort

 The purpose of the risk-based approach is NOT to reduce

testing effort
 It is to focus the testing efforts on the areas of highest risk –
this has been shown to significantly reduce problems in the
operational system for no increase (or even reduction) in
validation costs

©2015 Waters Corporation 15

 Risk Assessment – the process

©2015 Waters Corporation 16

 Responsibilities

The Customer
 MUST decide the high, medium, low ratings for each
requirement
 Has final responsibility for compliance (under law)
 Has to defend the risk assessment and validation approach
during an inspection

 Supplier or consultants can facilitate the meeting and suggest

risk scenarios for evaluation
 They must not contribute to the rating

©2015 Waters Corporation 17

 A Risk-Based approach
means some functions
may not be tested at all

©2015 Waters Corporation 18

 Case Study System

 Customer X is
implementing a Processing
Clients
networked
chromatography data
system in their QC lab.
 It will store all of their Database
HPLC/MS data from Server
the lab.

Acquisition
Clients

HPLCs / MS

©2015 Waters Corporation 19

 Risk-Based Approach CDS Case Study
Not directly impacted, as the product is
Assess GxP Product Quality produced upstream of the Empower 3 CDS
System
Impact
May be directly impacted by the Empower 3 END
Patient Safety CDS, as the assays controlled by the CDS will
produce GxP critical data. The assays and
instrument control are out of scope for CSV

Data Integrity Is highly impacted by the Empower 3 CDS

Limits &
boundaries
High Risk tested?
Risk Priority
Assessment Covered in
Create test in
standard
customized
testing
Medium Risk OQ
Priority

Low Risk Priority END

©2015 Waters Corporation 20

 A Risk-Based approach
means less testing
overall during validation

©2015 Waters Corporation 21

 Risk Assessment is Crucial
Project Planning Summary report Validation Summary
Initial Definition details any Report
deviations and
Definition is expanded residual
to capture risks
requirements

User Requirements Validation

Specification Plan

Functions from the URS are

assessed in the Risk
Assessment

Risk Assessment

Controls to reduce risk Functions identified as

are included in the SDS, high risk priority are
and pass into the System given focus in the
Configuration Test Plan
Design & Detailed Test Plan Verification
Configuration Activities
Specification
Verification activities
challenge the
risk-reduction
controls
Configuration

©2015 Waters Corporation 22

 Who should participate?

 Project leader
 System owner
 Department manager
 System users
 IT representative
 Quality representative
 Vendor representative
 Validation Subject Matter Expert

©2015 Waters Corporation 23

 ALL views are important

©2015 Waters Corporation 24

 Evaluation Process

©2015 Waters Corporation 25

 Risk Assessment Method

©2015 Waters Corporation 26

 Severity of Harm

 What is the harm?

 Potential harm should be identified based on hazards. Examples
of potential harm include:
- production of adulterated product caused by the failure of a
computerized system
- failure of an instrument at a clinical site that leads to inaccurate
clinical study conclusions
- failure of a computerized system used to assess a toxicology study
that leads to incomplete understanding of a drug’s toxicological
profile

©2015 Waters Corporation 27

 Probability of Occurrence

 What is the probability of a failure?

 Understanding the probability of a failure occurring to a
computerized system assists with the selection of appropriate
controls to manage the identified risks. For some types of
failure such as software failure, however, it may be very difficult
to assign such a value, thus precluding the use of probability in
quantitative risk assessments.

©2015 Waters Corporation 28

 Likelihood of Detection

 What is the detectability of a failure?

 Understanding the detectability of a failure also assists with the
selection of appropriate controls to manage the identified risks.
Failures may be detected automatically by the system or by
manual methods. Detection is useful only if it occurs before the
consequences of the failure cause harm to patient safety,
product quality, or data integrity.

©2015 Waters Corporation 29

 Following Critical Functions through the
Process

©2015 Waters Corporation 30

 Overall Risk-Based Validation
Approach

©2015 Waters Corporation 31

 Real Life Risk Assessment

Leading to this entry in the Test Plan…

©2015 Waters Corporation 32

 Customized OQ Tests

©2015 Waters Corporation 33

 Customized OQ Tests (continued)

©2015 Waters Corporation 34

 Customized OQ Tests (continued)

©2015 Waters Corporation 35

 Traceability Matrix

Note: this customer only wanted URS to testing documents mapped in their TM.
Normally it would also include DCS and FS (if available).

©2015 Waters Corporation 36

 Validation Summary Report

©2015 Waters Corporation 37

 Low Priority Function

©2015 Waters Corporation 38

 Following the Requirement

Risk Assessment

Test Plan

Traceability Matrix

©2015 Waters Corporation 39

 Validation Summary Report

©2015 Waters Corporation 40

 Summary

©2015 Waters Corporation 41

 Summary

 The examples given here are from a real Waters customer. The
customer engaged Waters to execute the complete validation
life cycle in cooperation with the customer’s end users and
system administrator.
 This customer was recently inspected by US FDA, who had no
observations on this validation of their Empower Enterprise
system
 A Risk-Based Approach is desired and encouraged by the
regulators
 It focuses effort where it’s needed
 It makes testing of a complex system manageable

©2015 Waters Corporation 42

 Risk-Based Exercise

©2015 Waters Corporation 43

 Case Study System

 URS-1: The password expiry interval

shall be configurable by an Processing
Clients
authorized user with that privilege
 URS-2: The system shall flag up any
result based on manual integration
 URS-3: An independent audit trail Database
Server
shall be maintained on all user
actions that create, modify or delete
data Acquisition
 URS-4: The system shall be able to Clients
search and retrieve data, using key
fields e.g. sample ID, user, date as
the search criteria
 URS-5: Data collected by the HPLCs / MS
acquisition clients must not be lost in
the event of a network disruption

©2015 Waters Corporation 44

 Exercise Part 1 – defining risk terms

Attribute Low Medium High

Severity of
Harm

Probability of
Occurrence

Likelihood of
Detection

Risk Priority
(overall rating)

©2015 Waters Corporation 45

 Exercise Part 2

Part 2: Risk Assessment

For each requirement, work out a risk scenario (what happens if the function
fails) and then assign ratings.

Use the GAMP figure M3.5 to

determine the Risk Priority for
each URS requirement.

©2015 Waters Corporation 46

 Exercise Part 3

Part 3: Controls / Configuration

For each URS requirement, analyse the controls that may be needed and any
configuration settings that would be recommended.

URS-1: The password expiry interval shall be configurable by an authorized user

with that privilege

©2015 Waters Corporation 47

 Exercise Part 4

Part 4: Test Plan

Outline test cases for each requirement, based on the assigned Risk Priority.

©2015 Waters Corporation 48

©2015 Waters Corporation 49