SCOPE Work Package 5

Signal Management

Best Practice
Guide

June 2016
SCOPE Work Package 5
Signal Management
Best Practice Guide

Contents
Authors 3

1. Introduction 4
1.1 Purpose of the document 4
1.2 Definitions and abbreviations 5
1.3 Background 7
1.4 Context of the Best Practice Guide 9
1.5 Findings from the survey on signal management 9

2. Methodology 11

3. Introduction to signal management 12

3.1 General context 12
3.2 Signal management processes at Member State level vs.
centralised European level 14
3.3 Role of PRAC in signal management 17

4. Recommendations 19
4.1 General recommendations 19
4.2 Recommendations related to signal detection 20
4.3 Recommendations related to signal prioritisation 21
4.4 Recommendations related to validation and assessment 21
4.5 Recommendations related to reports of special interest 22

5. Best Practice Guidance 24

5.1 Tracking of signals 24
5.2 Access to relevant information for signal management process 25
5.3 Signal detection 33
5.4 Signal validation 36
5.5 Signal prioritisation 39
5.6 Signal assessment 40
5.7 Reports of special interest 42
5.8 Data quality 47
5.9 Training 48

6. Conclusions 50

7. Annexes 51
Annex 1. SCOPE WP5 recommendations for future research and development 51
Annex 2. Signal validation and assessment checklist 55
Annex 3. Minimum information required for tracking of signals within the MSs 59
Annex 4. Signal detection methods and thresholds used in various NCAs 60
Annex 5. Regulatory Pharmacovigilance Prioritisation System 61
Annex 6. Glossary 62

8. References 65

2
SCOPE Work Package 5
Signal Management – Best Practice Guide

Authors

Alexandra Pacurariu, Anja van Haren, Anna-Lena Berggren, Birgitta Grundmark, Daniel Zondag,
Helle Harder, Mona Vestergaard Laursen, Phil Tregunno, Pilar Rayon, Rebecca Owen,
Sabine Straus

3
SCOPE Work Package 5
Signal Management – Best Practice Guide

1. Introduction

1.1 Purpose of the document

The Strengthening Collaboration for Operating Pharmacovigilance in Europe (SCOPE) Joint Ac-
tion has been established to maximise the effective implementation of the European Pharma-
covigilance legislation that came into effect in June 2012 by the National Competent Authorities
(NCAs) in European Union (EU) Member States (MSs). Funded by the Consumers, Health and
Food Executive Agency (CHAFEA), with contributions from the MS partners, SCOPE aims to
provide practical tools and guidance. SCOPE aims to deliver sustainable outcomes for MSs that
last beyond the end of the three-year project through the creation of training materials, living
documents and templates, which can be reviewed and adapted periodically.

SCOPE is divided into eight separate Work Packages (WPs). Three of these are focused on more
practical aspects – coordination, dissemination and evaluation – spanning all areas of the project.
The other five WPs focus on pharmacovigilance topics to deliver specific and measureable ob-
jectives, ranging from improvements in Adverse Drug Reaction (ADR) reporting to assessment of
quality management systems for NCAs.

SCOPE Work Package 5 (WP5) seeks to further improve signal management within the network
of EU Medicines NCAs.

Within WP5, the partners have been working on four topics:

1. Signal detection – lead: Medicines Evaluation Board (MEB).

2. Signal validation and prioritisation – lead: Agencia Española de Medicamentos y Productos

Sanitarios (AEMPS).

3. Signal confirmation and assessment – lead: Danish Medicines Agency (DKMA).

4. Reports of special interest – lead: Medicines and Healthcare Products Regulatory Agency
(MHRA).

Additionally, the Swedish Medical Products Agency (MPA) was actively involved in WP5.

Questionnaires have been used to gather information on all aspects of signal management: signal
detection, validation, prioritisation, confirmation, assessment and management of signals raised
from reports of special interest, including medication error and misuse/abuse of medicines. Using
the available data and information, this best practice guidance has been developed for the Euro-
pean Network and will be complemented with training sessions. The overall objective of the ac-
tivities of WP5 are to implement a shared understanding of best practice in signal management
across the EU network, in order to document and deliver recommendations regarding the timely
detection, management and assessment of safety signals across the EU network. There is a focus
on signals of special interest and medication error.

In addition, SCOPE WP5 identified areas for future research and development.

4
SCOPE Work Package 5
Signal Management – Best Practice Guide

1.2 Definitions and abbreviations

Terminology Description

ADR Adverse Drug Reaction

AE Adverse Event
AEMPS Agencia Española de Medicamentos y Productos Sanitarios
ATC Anatomic Therapeutic Chemical Classification
BPG Best Practice Guide
CAP Centrally Authorised Product
CIOMS Council for International Organizations of Medical Sciences
CHAFEA Consumer, Health and Food Executive Agency
CHMP Committee for Medicinal Products for Human Use
CMDh Coordination Group for Mutual Recognition and Decentralised Procedures –
Human
CPRD Clinical Practice Research Datalink
CSP Core Safety Profile
DKMA Danish Medicines Agency
DHPC Direct Healthcare Professional Communication
DME Designated Medical Event
EBGM Empirical Bayes Geometric Mean
EC European Commission
EMA European Medicines Agency
ENCePP European Network of Centres for Pharmacoepidemiology and
Pharmacovigilance
EPITT European Pharmacovigilance Issues Tracking Tool
eRMR electronic Reaction Monitoring Report
EU European Union
EUNTC EU Network Training Centre
EVDAS EudraVigilance Data Analysis System
FAERS FDA Adverse Event Reporting System
FDA Food and Drug Administration
GAB General Advisory Board
GVP Guideline on good pharmacovigilance practices

5
SCOPE Work Package 5
Signal Management – Best Practice Guide

Terminology Description

HLGT High Level Group Term

HLT High Level Term
IC Information Component
ICSR Individual Case Safety Report
IME Important Medical Event
IMI Innovative Medicines Initiative
IR Implementing Regulation
IRN Incident Review Network
LMS Lead Member State
MAH Marketing Authorisation Holder
MEB Medicines Evaluation Board
MedDRA Medical Dictionary for Regulatory Activities
MHRA Medicines and Healthcare products Regulatory Agency
MPA Medical Products Agency
MS Member State
NAP Nationally Authorised Product
NCA National Competent Authority
NUI Non Urgent Information
PASS Post Authorisation Safety Study
PRAC Pharmacovigilance Risk Assessment Committee
PRR Proportional Reporting Ratio
PSUR Periodic Safety Update Report
PT Preferred Term
PROTECT Pharmacoepidemiological Research on Outcomes of Therapeutics
RA Rapid Alert
RMP Risk Management Plan
ROR Reporting Odds Ratio
RPPS Regulatory Pharmacovigilance Prioritisation System
SCOPE Strengthening Collaboration for Operating Pharmacovigilance in Europe
SmPC Summary of Product Characteristics

6
SCOPE Work Package 5
Signal Management – Best Practice Guide

Terminology Description

SMART Signal Management Review Technical Working Group

SMQ Standardised MedDRA Query
SOC System Organ Class
SOP Standard Operating Procedure
WHO World Health Organization
WP Work Package
WPL Work Package Lead

1.3 Background
The objectives of WP5 of the SCOPE Joint Action were defined at the start of the project as:

‘to create, document and deliver recommendations for consistent and timely procedures for the
timely management of safety signals across the EU network. There is an additional focus on
detection and management of signals of special interest, such as those arising from medication
errors, drug abuse, misuse, off label use and from the use of biological medicines.’

Signal management is a key activity in pharmacovigilance. The signal management process is a

set of activities performed to determine whether, based on an examination of individual case
safety reports (ICSRs), aggregated data from active surveillance systems or studies, literature
information or other data sources, there are new risks associated with an active substance or a
medicinal product or whether risks have changed (1).

The EU legislation, implemented in 2012, introduced an EU-wide process for signal management
overseen by the Pharmacovigilance Risk Assessment Committee (PRAC) with specific responsi-
bilities and interactions between all stakeholders involved.

The EU legislation that includes provisions for signal management comprises:

 Regulation (EU) No 1235/2010 amending Regulation (EC) No 726/2004(2)

 Directive 2010/84/EU amending Directive 2001/83/EC (3)

 Commission Implementing Regulation (EU) No 520/2012 on the Performance of Pharma-

covigilance Activities Provided for in Regulation (EC) No 726/2004 and Directive
2001/83/EC (4)

7
SCOPE Work Package 5
Signal Management – Best Practice Guide

This legislation is supported by the ‘Guideline on good pharmacovigilance practices (GVP) Mod-
ule IX Signal management’ (5) and a European Medicines Agency (EMA) Questions and Answers
document (6). This Best Practice Guidance (BPG) is based on the guidance applicable at the time
of finalisation of this document and any reference in this document to GVP Module IX refers to
the version dated 22nd June 2012. Of note, GVP Module IX is under revision at the time of fina-
lisation of this document. This revision will provide more streamlined information on scientific
aspects of signal management as well as clarifications on terminology, roles and responsibilities
and processes, updated guidance on the periodicity of monitoring of EudraVigilance, procedural
options for signals validated by marketing authorisation holders, revised definitions and pro-
cesses for emerging safety issues. Also, an addendum on methodological aspects of signal de-
tection from spontaneous reports has been developed by the EMA that provides useful infor-
mation.

The pharmacovigilance legislation (2012) has had an impact on already existing systems at na-
tional level. A way to facilitate signal management, as described in the legislation, is to further
develop a common understanding and tools that could be used at the level of NCAs.

This BPG is a deliverable of WP5, together with a training plan covering signal management, with
a primary focus on the process in the MSs. It provides recommendations for efficient and effec-
tive signal management at MS level. The majority of these recommendations are based upon the
results of a survey conducted amongst MSs, during which best practices were identified (7).
Recommendations are also supported by literature and expertise within WP5, the SCOPE project
team, the SCOPE General Advisory Board (GAB) and the EMA.

The BPG builds upon the Report of the Council for International Organisations of Medical Sci-
ences (CIOMS) Working group VIII ‘Practical Aspects of Signal Detection in Pharmacovigilance’,
which contextualises different signal detection approaches within a holistic signal management
perspective (8).

At the level of the NCAs there are differences in terms of structure, organisation and resources
available. The recommendations from this BPG, together with the legal requirements on signal
management and the existing guidance (GVP) aim to facilitate current signal management in the
EU.

In addition to practical recommendations, SCOPE WP5 also provides recommendations for fu-
ture research and development (see Annex 1), aimed at improving signal management within the
EU regulatory network.

8
SCOPE Work Package 5
Signal Management – Best Practice Guide

1.4 Context of the Best Practice Guide

The target audience for this BPG is primarily European assessors and signal management staff
working at European NCAs involved in any step of the signal management process. The BPG is
also useful for other NCA staff, such as policy makers from national agencies or regional phar-
macovigilance centres, as applicable. It may also be useful for other stakeholders involved in
signal management, such as the EMA or Marketing Authorisation Holders (MAHs).

The BPG does not replace existing guidance, but provides a focus on the practical aspects and
challenges of signal management at the level of NCAs. The readers of the BPG should be familiar
with all relevant legislation and available guidelines, such as the GVPs Module IX – Signal man-
agement (5) and Questions and Answers on signal management (6). A basic understanding of
other regulatory procedures, such as Periodic Safety Update Reports (PSURs) and Risk Man-
agement Plans (RMPs) assessments and (safety) variations, is also recommended.

The most important aspect to keep in mind in the area of signal detection and signal management
is to apply sound scientific and clinical judgment. The BPG aims to help with the legislative con-
text. In the development of this BPG, SCOPE WP5 liaised with other initiative groups including:

 The Signal Management Review Technical Working Group (SMART), which is a collaboration
between MSs and the EMA to establish, disseminate and periodically review tools and meth-
odologies to facilitate the implementation of the signal management process

 The Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consor-

tium (PROTECT) project (9), which aimed to develop new methods and assess existing ones
for signal detection from spontaneous reports, electronic health records and clinical trials. The
Innovative Medicines Initiative (IMI) PROTECT group has developed a set of recommendations
pertaining to good signal detection practices, which were recently published (10)

 The SCOPE General Advisory Board (GAB), which ensured the liaison with the World Health
Organization (WHO), the EMA, healthcare professionals and patient organisations.

EU procedures for signal management (including information for MAHs) are provided in the GVP
Module IX.

1.5 Findings from the survey on signal management

The data used for the development of this BPG were collected through a web-based survey
among the MSs on current practices in signal management (with a focus on national activities).
The survey was conducted during the period of July 2014 to October 2014. Twenty-five out of
thirty-one MSs responded to the questionnaire.

The survey identified several challenges for the MSs, including:

 Ambiguity in terminology and definitions in signal management

 The need for resources and good training

9
SCOPE Work Package 5
Signal Management – Best Practice Guide

 The need to be aware of the available sources of information and access to data

 The need for tools to support the signal management process.

A summary of the survey findings is provided below, the full text can be found on the SCOPE
website: www.scopejointaction.eu/_assets/files/SCOPE-WP5-FULL-report.pdf.

Signal detection
MSs have systems in place to monitor ADRs and perform signal detection in national databases
as well as in EudraVigilance. Methods for signal detection in the national databases vary between
MSs. The screening of EudraVigilance takes place via the electronic reactions monitoring report
(eRMR) tool as provided by the EMA and this was identified as a time-consuming activity.

Signal validation
The majority of MSs have a signal validation step in place. The resources needed for the valida-
tion step differ by MS. Challenges identified in relation to signal validation are: limited access to
documents, limited information included in ICSRs, lack of resources, lack of expertise of asses-
sors and the definitions in the GVP Module IX.

Signal prioritisation
Prioritisation is a step in signal management and is performed by MSs throughout the whole
signal management process. Some MSs have a structured process in place. In the survey, a wish
for more guidance on how to prioritise signals was identified.

Signal confirmation
The survey indicated that signal confirmation, as it is described in the GVP Module IX (version 1),
creates confusion with regards to the difference between a validated signal and a confirmed
signal. The way the process has been divided in the legislation and the GVP creates challenges
in interpretation.

Signal assessment
Separating the signal validation and assessment steps is not easy and uncertainty on how to
ensure the right focus during the different steps in signal management was identified as a chal-
lenge. Compiling all relevant data as well as access to (exposure) data was identified as complex.
For the signal assessment, lack of human resources and a need for further training of assessors
were also identified as important aspects.

Reports of special interest

With regards to reports of special interest, the survey showed that limited strategies were imple-
mented for signal detection in special populations such as children and elderly and for vaccines,
biologicals, medication errors, occupational exposure, medication abuse, etc. A need to improve
knowledge and further training of assessors in this field was identified.

10
SCOPE Work Package 5
Signal Management – Best Practice Guide

2. Methodology

This BPG is based on the outcome of the cross-sectional survey. It reflects the situation,
knowledge and guidance at that moment in time (October 2014). The data from the survey was
analysed and discussed by SCOPE WP5 partners. To identify best practices, the results of the
survey and a variety of information sources were used, including feedback from experts, the
knowledge within WP5, review of literature and available guidance. A literature review (7) was
performed by WP5 to further enrich the BPG.

The draft BPG was reviewed by the SCOPE Work Package Leads (WPL), the SCOPE General
Advisory Board (GAB) and the EMA, and comments were received during pilot training from as-
sessors; feedback received was used to optimise the BPG.

Processes within NCAs are organised differently across the EU, taking into account factors such
as the size of databases and resources available, and it is therefore expected that signal man-
agement processes will differ between MSs. Consequently, not all recommendations will be ap-
plicable or relevant to the same extent for every MS.

11
SCOPE Work Package 5
Signal Management – Best Practice Guide

3. Introduction to signal management

3.1 General context

The term ‘signal’ has been used widely in pharmacovigilance and the following three aspects are
considered important to better understand the context of the signal management process:

1. The starting point for signal detection is that all activities take place in the presence of
some level of uncertainty and there is always a need to apply sound judgement. Signals
have different degrees of probability according to the completeness and strength of the rele-
vant data assessed. Evidence may be found in different sources, be of differing strengths and
can be accumulated or gathered over time. Meyboom et al (11) stated that ‘A signal in
pharmacovigilance is more than just a statistical association. It consists of a hypothesis, to-
gether with data and arguments, arguments in favour and against the hypothesis. These relate
to numbers of cases, statistics, clinical medicine, pharmacology (kinetics, actions, previous
knowledge) and epidemiology, and may also refer to findings with an experimental character.’
This is not, strictly speaking, a definition, but a description. One useful aspect of this descrip-
tion is that it emphasises that signal detection incorporates clinical judgement that goes be-
yond an automatic process.

2. The element of ‘suggests’ in the legal definition of a signal emphasises that a signal is
basically a hypothesis and does not necessarily translate in all cases into a definite
causal association (IR, Art 19: ’signal’ means information arising from one or multiple
sources, including observations and experiments, which suggests a new potentially causal
association, or a new aspect of a known association between an intervention and an event
or set of related events, either adverse or beneficial, which is judged to be of sufficient
likelihood to justify verificatory action).

Example
The EMA 2015 Annual Report on EudraVigilance (12) shows that during 2015, the EMA
Signal Management Team reviewed in total 2372 potential signals (i.e. drug-event pairs
from screening of the EudraVigilance database, medical literature, information from other
regulatory authorities with the potential of being a safety issue). Of these, 61 signals validated by
the EMA ended up on the PRAC agenda in 2015.

3. Not all signals represent risks and not all signals will require an additional formal regu-
latory action (e.g. update of the Summary of Product Characteristics (SmPC)) after assess-
ment has been performed.

12
SCOPE Work Package 5
Signal Management – Best Practice Guide

Example
The EMA 2015 Annual Report on EudraVigilance (12) shows that the PRAC prioritised
and assessed 102 signals during 2015. 33% of the signals resulted in a recommen-
dation for an update of the product information (PI), including the distribution of a Direct
Healthcare Professional Communication (DHPC) on four occasions to highlight important new
safety information to healthcare professionals. Twenty-seven signals (26%) were closed and sub-
ject to routine safety monitoring. Four signals resulted in a recommendation to update the RMP;
another signal was further assessed through a Post Authorisation Safety Study (PASS), and one
signal was evaluated in a referral procedure. The evaluation of 35 signals (35%) was ongoing,
including 20 via a follow-up signal procedure and 15 in the next PSUR.

Figure 1. Outcomes of PRAC signal assessments in 2015

EMA 2015 Annual Report on EudraVigilance (12)

13
SCOPE Work Package 5
Signal Management – Best Practice Guide

3.2 Signal management processes at Member State level

vs. centralised European level
The 2012 EU legislation introduced legal requirements for (EU) signal management. The most
detailed part of the legislation is provided by the Implementing Regulation (IR), which describes
activities in detail and assigns responsibilities to different stakeholders (4). As the role and tasks
of the PRAC in the EU signal management process are clearly described in GVP Module IX, this
BPG focuses on the signal management process within a MS and their interaction with the EU.

The IR also defines that signal management consists of the following activities:

 Signal detection

 Signal validation

 Signal confirmation

 Signal analysis and prioritisation

 Signal assessment

 Recommendation for action.

For some of these activities, the IR provides definitions or explanations, whereas others are ex-
plained in the GVP Module IX and/or in the EMA’s Questions and Answers document on signal
management (6). It should be noted that the terminology used in the EU signal management
process is different from the terminology used commonly in scientific literature (8,13). For exam-
ple, in scientific articles the term ‘verification’ of a signal is often used and not the term validation.
The CIOMS VIII guidance, for example, defines the term ‘signal verified’ as ‘a signal of suspected
causality, that has been verified either by its nature or source or by formal verification studies.’

Signal management activities are often described as a linear process (as shown in Figure 2) con-
sisting of different steps to be completed in a sequential order (as is the case in the IR for exam-
ple), but it is challenging to separate the different steps and not feasible to assign a time sequence
or order in which they should be completed. It is acknowledged in GVP Module IX that flexibility
may be required during the whole process.

The role of different stakeholders throughout the signal management process as described in the
current legislation is shown in the flowchart in Figure 2.

14
SCOPE Work Package 5
Signal Management – Best Practice Guide

Figure 2. Signal management process/responsibilities and activities for regulators as

provided in the legislation

MSs use these concepts in their national signal management process. However, some differ-
ences between MSs could be identified based on the MSs’ responses to the SCOPE WP5 survey:

1. Some MSs perform the signal detection and validation at national level and then subsequently
enter the signal into the European Pharmacovigilance Issues Tracking Tool (EPITT) (as shown
in Figure 2).

2. Other MSs add an additional decision-making loop before entering a signal into EPITT (as
shown in Figure 3). In this way, a signal may be reviewed in greater depth at national level
before the NCA enters the signal in EPITT.

Figure 3. Signal management process as applied at MS level with an additional

decision-making loop

In this (optional) additional decision-making loop at national level, differences between MSs in
signal validation may originate. Depending on the organisation and the resources in a MS, the
validation of the signal might be more extensive at national level. For example, a more elaborate
evaluation of data might already have taken place at national level before next steps are taken.

Furthermore, in some MSs, national and/or regional pharmacovigilance centres might be involved
in the signal management process and, if that is the case, multiple decision-making loops may
be in place (as shown in Figure 4).

15
SCOPE Work Package 5
Signal Management – Best Practice Guide

Figure 4. Interaction between national / regional pharmacovigilance centres, MSs and PRAC

Consequently, the different organisational structures of MSs might lead to different ways of deal-
ing with a signal at the national level and to different timing in entering a signal in EPITT.

The SCOPE WP5 survey identified that MSs may find it challenging to apply signal management
concepts/steps defined in the legislation and the GVP (e.g. validation, confirmation, and prioriti-
sation) within their national organisations, especially for those steps that are not included or may
have a different understanding in the scientific literature. In some MSs CIOMS VIII guidance and
terminology (8) is used in the development of national processes, before the legislation came into
force. This might pose challenges, for example in line with the CIOMS VIII guidance, a signal
management standard operating procedure (SOP) at the national level might describe ‘how signal
prioritisation and evaluation are approached’, whereas in the EU legislation prioritisation is as-
signed as a responsibility to the PRAC. However, for MSs, prioritisation is a continuous process
that is considered at every step of signal management. The EU legislation and GVP guidance
describe concepts that apply specifically to procedures at the central EU level, however these
might not always be applicable/feasible at national level. Within the EU signal management pro-
cess, all actions lead to entering a signal into EPITT, but the process that precedes this might
differ between MSs as well as between the EMA and MSs. A signal at national level might be
reviewed at several different steps (in line with national signal management SOP), before the NCA
decides to enter the signal into EPITT.

16
SCOPE Work Package 5
Signal Management – Best Practice Guide

For their national signal management SOP, MSs, depending on their organisation, can apply sig-
nal management definitions and processes or steps that are different from the EU legislation and
guidance, as shown in Figure 5. Once a signal is entered into EPITT, definitions provided in the
IR and explanations or clarifications provided in GVP Module IX are applicable. However, for the
signal management steps that take place at a national level, other working concepts can be more
suitable, for example those provided in CIOMS VIII guidance (8).

Figure 5. Various stages of signal management process and the various definitions used
according to the level where signal management is performed (EU vs national)

3.3 Role of PRAC in signal management

As mentioned in the previous section, this BPG focuses on the national signal management pro-
cesses in the MSs. As both processes, national as well as central EU, are part of the overall signal
management process, a brief summary of the process at the EU level is included here; for full
information on the central EU level, please refer to GVP Module IX (5) and the Questions and
Answers document (6).

The PRAC at the EMA has a central role in scientific assessment and decision-making in relation
to signal management.

The PRAC shall regularly review the methodologies used for determining the evidentiary value of
a signal and publish recommendations, as appropriate (IR, Art 20(3)). According to the legislation,
the evidentiary value of a signal shall be determined by using a recognised methodology taking
into account the clinical relevance, quantitative strength of the association, the consistency of
the data, the exposure–response relationship, the biological plausibility, experimental findings,
possible analogies and the nature and quality of the data (IR Art 20(1)).

17
SCOPE Work Package 5
Signal Management – Best Practice Guide

The legislation puts specific obligations on MAHs1, NCAs and the EMA for the monitoring of the
EudraVigilance database, and on NCAs for monitoring data originating in their territory (1-4, see
also section 5.3 Signal Detection). Once the signal identifier – either the NCA or the EMA – has
decided that a signal should be entered in EPITT, it can be confirmed or not by the Lead Member
State (LMS) or PRAC Rapporteur. If the signal is confirmed, the signal will be discussed during
the next plenary PRAC meeting. The PRAC may further amend the scope of the signal manage-
ment by extending it to other active substances of the same class of medicinal products or to
other related adverse reactions. When further assessment is considered needed within the signal
management procedure, the PRAC will nominate a Rapporteur and define a timeframe for the
assessment. After signal confirmation, PRAC decides on the prioritisation, leads on the assess-
ment of the data and provides recommendations regarding the regulatory actions to be taken.
The PRAC ensures that new or changed safety issues are translated into regulatory actions, as
appropriate.

After discussion at the plenary meeting, the PRAC recommendation may include any or a com-
bination of the following conclusions (6):

 No need for further evaluation or action at this point in time

 Need for additional information – e.g. via a cumulative review or in the next PSUR

 Need for regulatory action – e.g. update of the SmPC to reflect the findings, RMP update,
start of a referral, etc.

Actions recommended by PRAC may be accompanied by additional communication measures,

e.g. a Direct Healthcare Professional Communication (DHPC).

If based on the data available, a need for immediate action before the upcoming PRAC meeting
is identified, an NCA can act at national level and take temporary measures and trigger at the
same time a Rapid Alert (RA) and a review by the Incident Review Network (IRN). This virtual
group reviews incidents from a managerial perspective in terms of their impact on public health
and the measures needed to address them. This process is explained in the incident manage-
ment plan (14), which aims to ensure that concerned bodies in the EU take appropriate action
whenever incidents (new events or information) arise concerning human medicines. The activities
in the plan and of this managerial group focus on managing incidents and do not replace the
work of the PRAC, the Committee for Medicinal Products for Human Use (CHMP), or the Coor-
dination Group for Mutual Recognition and Decentralised Procedures – Human (CMD(h)) (14).

1 Signals identified by MAHs are outside the scope of this BPG, see GVP IX for more information on MAHs’
obligations.

18
SCOPE Work Package 5
Signal Management – Best Practice Guide

4. Recommendations

This section of the BPG presents the SCOPE WP5 recommendations on signal management. An
outline of the recommendations is presented in chapter 4, followed by a more thorough best
practice guidance in chapter 5 to develop further understanding of the recommendations.

4.1 General recommendations

These recommendations refer to general areas in signal management, such as access to data,
exchange of information and tracking of signals, which are not related to any specific steps in
the signal management process.

4.1.1 For signal management activities, access to data is essential and this access should
be facilitated. Access to all relevant documents and data sources is essential to pharma-
covigilance assessors involved in signal management to ensure thorough assessment.
Some of the documents are publicly available, but not all are easily accessible within all
NCAs. Some of the most relevant documents and data sources are clinical trial data, prod-
uct application dossier, SmPC, patient leaflet, PSURs, RMPs, EPITT, decision-making
documents (e.g. from scientific committees or regulatory procedures), scientific literature,
data provided by an MAH and data sources outside spontaneous reports, such as regis-
tries and databases, etc. MSs should ensure that all relevant information is available for
assessors and this includes relevant documents stored within the EMA domain.

4.1.2 The EU network would benefit from the possibility of an early exchange of information
on signals in order to inform other MSs and prevent duplication of work. Given there
are no tools currently available for this specific purpose, there are a number of ways to
facilitate this exchange of information: inform the Lead Member State (LMS) by e-mail,
distribute a non-urgent information request (NUI), or enter the signal in EPITT and propose
to the LMS to not confirm this signal (meaning that the signal will not be included in the
PRAC agenda). In future, this sharing should be facilitated by the creation of a new signal
management tool (see Annex 1).

4.1.3 To ease traceability of how signals are detected, validated and assessed, MSs would
benefit from a minimum set of variables to be tracked at MS level. Tracking of signals
in a sufficient way at MS level can be a challenge, and various systems and routines are in
place across MSs. Based on the information provided in the survey, a minimum set of
variables to be tracked at national level was identified.

19
SCOPE Work Package 5
Signal Management – Best Practice Guide

4.2 Recommendations related to signal detection

4.2.1 Existing heterogeneity in signal detection methods within the EU network is beneficial.
MSs implement different signal detection methods on different national datasets. The dif-
ferences in the national databases and the different methods applied can be considered a
strength of the system. In signal detection, it is important that the methods applied are
appropriate for the respective databases and there is no ‘one size fits all’ solution. Methods
that are appropriate for large international datasets would not be appropriate for smaller
national datasets. In addition, this diversity in databases and methodologies allows the
detection of different signals in the different databases.

4.2.2 Disproportionality methods should be applied to databases of appropriate size and

background. Use of disproportionality methods is not appropriate in all situations. Appli-
cation of disproportionality methods to a dataset that is too small or with a limited set of
drugs or events reported might not be relevant and provides no added benefits compared
to using qualitative methods and simple quantitative methods. Therefore, it may be more
appropriate for small databases to apply qualitative methods or simple rule-based meth-
ods (e.g. count of case reports) or a combination of these.

4.2.3 The method of disproportionality analysis is less important than the threshold for de-
tection chosen, which should be duly justified. It is important that any signal threshold
applied is appropriate to the size of the database, the products it contains, and the level
of sensitivity (how many signals will be identified) and expected precision (positive predic-
tive value, how many identified signals are real signals). This applies when complex dis-
proportionality methods or rule-based approaches (such as reviewing all fatal case reports)
are employed. It is important that the chosen approach can be shown to identify signals
through a validation/testing process (meaning that the statistical analyses should be robust
against spurious associations due to random variability). This process should ensure that
the approach can both detect issues that have been identified in the past for the products
under review, and also that it is no worse than the previous approach at identifying safety
issues of interest for review in a side-by-side comparison. This recommendation is based
on findings from the IMI PROTECT group; more detailed recommendations regarding the
choice of disproportionality method can be found in the Good Signal Detection Practices:
Evidence from IMI PROTECT (10).

4.2.4 Focusing signal detection on predefined events is helpful. Focusing the signal detection
efforts on serious events and events that are likely to have a high public health impact
(such as Important Medical Event (IME) and Designated Medical Event (DME) lists) is
recommended.

20
SCOPE Work Package 5
Signal Management – Best Practice Guide

4.2.5 In quantitative signal detection, analysis should be performed at the level of the stand-
ard Medical Dictionary for Regulatory Activities (MedDRA) preferred term (PT) level.
Screening at PT level is current practice and there are currently no data to suggest that
standard screening at higher aggregation levels (Higher Level Term (HLT), Higher Level
Group Term (HLGT) or Standardised MedDRA Query (SMQ)) would detect more signals or
deliver signals at a an earlier stage in time than when using Preferred Term (PT) level for
screening. This recommendation is based on findings from the IMI PROTECT group; more
detailed recommendations regarding the choice of level at which screening should be per-
formed can be found in Good Signal Detection Practices: Evidence from IMI PROTECT (10).

4.3 Recommendations related to signal prioritisation

4.3.1 Prioritisation should be a continuous process performed during the whole signal man-
agement process, rather than in a single step. While prioritisation should be systemati-
cally performed during the initial steps of signal management, it does not prevent further
prioritisation during the whole process. Signal prioritisation is required to focus the re-
sources available on the most important signals. A standardised approach based on the
most important criteria that can be used should be in place, and validated criteria should
be employed as much as possible. However, prioritisation could take into account different
variables depending on the step of the signal management process.

4.4 Recommendations related to validation and assessment

4.4.1 A validated signal should be differentiated from a confirmed signal. Signal confirma-
tion, based on the legal terminology, is a procedural step only relevant when a MS wants
to bring a signal to the PRAC agenda. At the EU regulatory level signal confirmation only
refers to EPITT. After a signal is entered in EPITT by the signal identifier, the LMS or Rap-
porteur can decide to either mark the signal as ‘confirmed’ or ‘non-confirmed’ within 30
days. After confirmation, the signal will be discussed at the next PRAC meeting for the
initial analysis and prioritisation according to the IR.

4.4.2 The processes of signal validation and signal assessment could benefit from a con-
sistent approach between different MSs and one way that could help to achieve this
is by using a checklist. The EU network would benefit from a consistent approach to
signal validation and assessment, and a tool such as a checklist could be used to achieve
this. A basic checklist that summarises the most important activities and relevant sources
of information in validation and assessment should be developed to meet this objective.
An example of such a signal validation and assessment checklist is provided in Annex 2.

21
SCOPE Work Package 5
Signal Management – Best Practice Guide

4.4.3 Multidisciplinary teams with experts from different areas and various levels of exper-
tise can be helpful and increase the quality of signal assessment. A multidisciplinary
team of experts from relevant areas can help provide thorough clinical statistical and phar-
maceutical expertise necessary to guarantee the quality of a signal assessment (8). This
can be achieved in different ways using both internal and external expertise and will de-
pend on the organisational structure and possibilities in each MS.

4.5 Recommendations related to reports of special interest

4.5.1 Awareness of methodologies and approaches for analysis of reports of special inter-
est must be raised. Very few MSs had methodologies in place to help identify signals
against all categories of reports of special interest, despite their scientific, public health
and political importance.

4.5.2 Where statistical analysis is applied, subgrouping into specific categories of interest
(e.g., paediatric reports, vaccine-related reports) may be a helpful approach for anal-
ysis of some reports of special interest. The IMI PROTECT group identified that sub-
grouping may improve both the sensitivity and precision of statistical methods (15). Fur-
ther, such analysis methods may serve to focus assessors’ attention on specific areas,
leading to higher quality signal detection in these groups. However, use of such ap-
proaches should only be considered if the database is of an appropriate size that allows
analyses to be statistically valid. This recommendation is based on findings from IMI
PROTECT group; more detailed recommendations regarding subgrouping and stratifica-
tion approaches can be found in Good Signal Detection Practices: Evidence from IMI
PROTECT (10)

4.5.3 Assessors involved in signal management should have knowledge of the guidance in
the Product/ Population Specific GVP Modules as they are made available. GVP P1 –
Vaccines for prophylaxis against infectious diseases (16) and GVP P2 – Biological medici-
nal products (17) and others under development provide helpful strategies for analysis of
these product groups.

4.5.4 Qualitative methods should be used for identification of medication errors. The survey
did not identify dedicated statistical methods proven in identification of such case reports
against the background noise in the data. Important case reports may not be highlighted
explicitly by reporters or coding personnel using relevant MedDRA terms, and numbers
may not be large enough for quantitative analysis.

22
SCOPE Work Package 5
Signal Management – Best Practice Guide

4.5.5 Signal detection activities for medication errors should focus on the identification of
harm as opposed to only coded medication error terms. Whilst the Medication Errors
SMQ may assist in identification of some relevant case reports, MSs should not recode
reaction terms based on their interpretation by adding medication error terms. Any addi-
tional terms thought suitable should be coded in sender diagnosis (E2B(R2) data element
B.5.3). Signal detection should focus on harms and understanding whether medication
error may have contributed to them.

4.5.6 MSs should build relationships between themselves and other national bodies with
responsibility for medication errors. As MSs may not be the primary receiver of medica-
tion error data, building relationships between other national bodies will assist in compre-
hension, detection and communication associated with medication errors.

4.5.7 Use of Designated Medical Events (DMEs) or equivalent lists should be considered
for analysis of specific reaction terms of interest. A number of MSs identified this ap-
proach as helpful in ensuring a focused review of serious events and those frequently as-
sociated with ADRs.

23
SCOPE Work Package 5
Signal Management – Best Practice Guide

5. Best Practice Guidance

In line with the above recommendations, this chapter presents selected topics of interest, where
more detail is given on the tasks for each signal management step, with a focus on national
processes (at MS level). The last two sub-chapters are more general and refer to the topics of
data quality and training.

Each section starts with a brief introduction where current legislation and guidance are presented
(if applicable), followed by the best practice.

5.1 Tracking of signals

The IR (4) requires NCAs to have an audit trail of their signal management activities in EudraVigi-
lance and for relevant queries and their outcomes, including how signals have been detected,
validated, confirmed and assessed. The GVP Module IX (5) has broadened the MS obligations by
stating that ‘all validation, prioritisation, assessment, timelines, decisions, actions, plans, report-
ing as well as all other key steps should be recorded and tracked systematically’.

Once a signal is entered in EPITT, the tracking in line with IR is ensured. Before that step, MSs
need to ensure adequate tracking of their signal management activities at the national level.

Best practice
Specific requirements for tracking tools at a national level are not described in EU
pharmacovigilance legislation or guidance documents. Therefore, it is up to individual
MSs to assess their own needs and implement a system that best meets their specific
situation and processes.

Example
Example of how different tracking systems can be used, depending on the step in the
signal management process.

A signal is detected when monitoring EudraVigilance via the eRMR. The ‘signal status’ col-
umn and ‘signal comments’ column in the eRMR are used to document the outcome of the vali-
dation step. The signal is subsequently discussed at internal meetings; the minutes of these
meetings are considered part of the signal management tracking system. Once the validated
signal is entered in EPITT, any subsequent steps are tracked from here onwards.

Whereas the tracking tools used may vary widely, based on the survey information MSs were in
agreement on the minimum information that should be collected for tracking. As a minimum, the
variables mentioned in Annex 3 should be tracked (see Annex 3).

24
SCOPE Work Package 5
Signal Management – Best Practice Guide

At the moment, retrieving information about signals followed up in other procedures is challeng-
ing, since no systematic tracking (tool) across procedures is in place. According to GVP Module
IX (5), tracking systems should also include signals for which the validation process did not sug-
gest a new potentially causal association, or a new aspect of a known association. MSs may use
the same systems for tracking both validated and non-validated signals.

It is considered good practice to have processes in place to ensure that non-validated signals
are not lost during follow-up. Such examples include:

 Discussion at a dedicated signal detection meeting for information purposes

 Regular review by a National Pharmacovigilance Committee

 Mandatory check of data related to previous non-validated signals during signal detection for
the same medicinal product.

The information should be recorded and archived.

5.2 Access to relevant information for signal management

process
Best practice
This section presents a general practical review of the most commonly used sources
of information for both the initial signal validation and the more exhaustive signal
assessment.

The evaluation of data supporting a signal is performed during almost every step of the signal
management process, from signal detection to signal assessment. However, it is important to
acknowledge that, the importance of each data source, the level of the data review and the aim
and focus of the evaluation may vary for each step. For example, during the validation of a signal,
the evaluation of the data should be aimed at deciding if further analysis is necessary and the
focus should be on determining if the signal reflects new information and if it is at least a reason-
able possibility. For signals originating from spontaneous reports, at a minimum, it should be
made sure that the signal is not only based on duplicate reports, and that there is plausible time
to onset. During signal assessment, the evaluation of the data is aimed at deciding if a regulatory
action is needed and the focus should be on reaching a final conclusion on the causal relationship
and considering the need for (additional) risk minimisation measures (see Figure 6).

25
SCOPE Work Package 5
Signal Management – Best Practice Guide

Figure 6. Aims of various signal management steps and how this might influence
data evaluation

Furthermore, the data relevant and available at each step may vary, depending on who performs
the evaluation (e.g. a different MS) or if additional information has been provided (e.g. by the MAH
as a result of a PRAC recommendation).

Signal validation is an important step in the signal management process. In the legislation, vali-
dation is discussed as a step in the signal management process that is separate from the as-
sessment of a signal. IR (Art 21) defines signal validation as the process of evaluating the data
supporting the detected signal in order to verify that the available documentation contains suffi-
cient evidence demonstrating the existence of a new potentially causal association, or a new
aspect of a known association, and therefore justifies further analysis of the signal. Signal vali-
dation will determine if a further assessment is warranted or if the signal could be refuted (further
assessment is not considered necessary, e.g. validation showed that the signal was based only
on duplicate reports).

All activities at every step in signal management take place in the presence of some uncertainty
(8) and any decision will need sound judgement.

When analysing the validated signal, NCAs and the EMA may take into account other information
available on the medicinal product. There are many useful data sources that assessors can review
once a signal is detected. Each data source might have different relevance depending on the
step of the signal management process (as shown in Figure 7).

26
SCOPE Work Package 5
Signal Management – Best Practice Guide

Figure 7. Sources of information and their relevance depending on the signal

management step

Practical advice for assessors on how to best use these sources of information in signal man-
agement is provided in the next section and it should be highlighted that in every step of the
process sound judgement should be applied for each decision taken.

Summary of Product Characteristics (SmPC)

The SmPC is an important source of information in signal validation to assess whether there is a
new potentially causal association, or a new aspect of a known association and, therefore,
whether the signal justifies further analysis. In order to evaluate this, the information already in-
cluded in the SmPC of the active substance should be taken into account. Only signals for which
there is no previous awareness and signals that may reflect a new aspect of a known association
(such as a change in frequency, temporal persistence, severity or a change in the outcome or
reported fatality rate) should be validated.

A signal can relate to an already known safety issue and not reflect new information and therefore
not fulfil the definition of a signal. It is important to check if the event is already reflected in the
SmPC of the product. However, there are two important points to acknowledge:

27
SCOPE Work Package 5
Signal Management – Best Practice Guide

a) Within a single MS, there may be differences between the SmPCs for a given active
substance. If an assessor notices that an event is mentioned in one SmPC, but not in another
(of the same active substance), the event would not fulfil the definition of a signal (it is not
new) and can be considered a national harmonisation issue.

b) There may be differences between SmPCs from different MSs for the same active sub-
stance and it is not practically feasible to check all SmPCs available in the EU. As a
consequence, it can happen that a MS validates and enters a signal into EPITT that appears
to be already reflected in the SmPC of another MS. It may occur that a signal discussed at
PRAC appears to be reflected in a SmPC somewhere in the EU. This cannot be avoided or
solved within SCOPE WP5. If a signal is marked as not-confirmed by a LMS or Rapporteur
with the justification that it is already reflected in a local SmPC, this could trigger other MSs
to initiate a harmonisation activity at national level.

While access to the currently approved SmPC of a centrally authorised product (CAP) is easy –
it is obtained via the EMA website and there is one single SmPC in the whole EU – checking the
SmPC of a nationally authorised product (NAP) in different MSs can be very challenging and is
not always feasible, particularly in view of language challenges. A SmPC for the same active
substance can differ not only across the different MSs, but also within a MS. The IMI PROTECT
group developed an SmPC dataset containing the minimum safety information for centrally au-
thorised products (the Adverse Drug Reaction Database) in order to check if a certain event is
reflected in the SmPC of a product (9).

Example
Checking if an event is reflected in the SmPC is not as straightforward as just checking
if it is included verbatim in section 4.8 of the SmPC of the active substance. An event
can be covered by similar (but different) PTs (e.g. anaphylactic shock by anaphylaxis; ver-
tigo covered by dizziness, swelling tongue covered by angioedema, etc.) or by a HLT. On the
other hand, an event might be reflected in the SmPC, but the signal might provide new additional
information (e.g. on further anatomical specification, severity or duration) and the event might not
be considered to be reflected when this specification is not specified (e.g. rash does cover mor-
billiform rash, but not Stevens Johnson syndrome; fulminant hepatitis should not be considered
to be reflected by liver injury). An event might also be reflected as a class effect, either as a
warning or in section 4.8. Expert medical judgement should always be used on a case-by-case
basis to determine if the signal reflects information not reflected by the current wording in the
SmPC and the need for further action in view of the information already included compared to
the information originating from the signal.

28
SCOPE Work Package 5
Signal Management – Best Practice Guide

In addition to section 4.8 of the SmPC, other relevant sections to check are section 4.3 (for any
contraindications), section 4.4 (for any warnings and precautions suggestive of the event), section
4.5 (for any information of a known interaction), and section 4.6 (in case of a signal related to
fertility, pregnancy and lactation). It should always be considered that, even when there is infor-
mation available in these sections, a signal might provide new information that can lead to an
update of the SmPC in the same or in a different section.

Regarding drug interactions, it is not always the case that there is reciprocity/mirroring of the
information in the SmPCs of both products. Only in the case of drug interactions that lead to a
contraindication is reciprocity considered essential. However, full consistency of reciprocal drug-
drug interactions information in SmPCs is a challenge and expert judgement should always be
used to consider if the signal provides new information on the safety issue.

A signal can also be validated if the available data suggests a change in the frequency of the
event reported in the SmPC (e.g. reports can suggest a higher frequency than expected), dura-
tion, severity, pattern (e.g. affecting a specific population) or a change in the previously reported
outcome (e.g. fatal cases).

Periodic Safety Update Report (PSUR)

PSURs (when available) are a relevant source of information and can be used if:

1. There is a PSUR assessment ongoing at the same time of the signal validation/ confir-
mation. In this case it is useful to check if the specific topic of the signal is already included
in the PSUR.
a) If it is already included, the signal could be considered to be addressed in the PSUR
assessment procedure and no further assessment might be warranted.
b) If the topic is not addressed, it could be considered as to whether to request a discussion
in the PSUR (if there is sufficient time for it given the timetable of the assessment); other-
wise, if the assessment is already ongoing and there is not sufficient time to address the
signal during the current procedure, it could be decided to proceed with addressing the
signal via the signal management procedure.

2. The submission date of the next PSUR is close. For practical reasons, if the date of sub-
mission of the next PSUR for the active substance is close enough, it might be decided to
address the signal in the next PSUR assessment.

Furthermore, the most recently available PSUR and its corresponding assessment report can
also be useful documents for the signal assessment (also, for example, to check exposure).

The main sections with information on signals are:

a) Section VII.B.5.15 ‘Overview of signals: new, ongoing or closed’. This can be checked if
the safety issue corresponding to the signal has been addressed.
b) Section VII.B.5.16 ‘Signal and risk evaluation’. This section gives information on the risks
that are addressed in the RMP of the active substance.

29
SCOPE Work Package 5
Signal Management – Best Practice Guide

Risk Management Plan (RMP)

RMPs could also be a useful source of information for the validation or assessment of a signal.
Especially interesting in this context are the safety specifications of the product and the pharma-
covigilance plan. Within the safety specifications of the product, the important identified risks
give information about safety issues for which there is sufficient evidence supporting a causal
association and are therefore considered to be already known for the product and also covered
in their SmPC.

Checking the important potential risks might be informative to assess if during the development
of the drug, the risk was identified as potential, and the data leading to the signal might provide
additional information on the probability of a possible causal association. For instance, for a sig-
nal for which the data supporting a causal association is considered rather weak, the fact that
the event is listed as an important potential risk in the RMP of the product might make the signal
stronger and help the assessor further in the decision-making process.

It might be helpful for the assessor to have knowledge on the safety profile of the product that
was identified during development of the RMP, such as the important identified risks, important
potential risks and missing information.

The pharmacovigilance plan in the RMP also provides information on any post-authorisation
safety studies that are planned or ongoing for the product and that might aim to further investi-
gate a safety issue. The information provided in this section of the RMP can also be helpful in
further managing the signal.

Since March 2014, the EMA has published summaries of risk management plans for centrally
authorised medicines on its website to increase transparency and public access to relevant in-
formation on medicines. The CMD(h) has also started a programme for publishing RMPs for na-
tionally authorised products when available (18).

EPITT
EPITT is a web-based system that supports the tracking of pharmacovigilance issues at a Euro-
pean level, including rapid alerts (RA), non-urgent information requests (NUIs) and signals. Sig-
nals that have been previously validated by the EMA or MSs are reflected in EPITT, where all
relevant information on the safety issue is kept (for non-confirmed signals, a justification for not
confirming is provided; for confirmed signals, the signal assessment report and PRAC recom-
mendations are provided). It is therefore a very useful source of information for checking any
previous awareness of a specific safety issue and any actions that might have been taken. It
would therefore be very beneficial for all assessors involved in signal management to have access
to EPITT.

30
SCOPE Work Package 5
Signal Management – Best Practice Guide

Ongoing variations
When assessing a signal, it is important to know if there is an ongoing variation that addresses
the same or a related safety issue. Sometimes a signal may appear at the same time that a vari-
ation procedure is addressing the same safety issue at a national level or in another MS. Being
aware of this can prevent duplication of work and spending resources on a safety issue that is
already addressed.

While this can be relatively easy to check for centrally authorised products or at a national level
for nationally authorised products (single national variations), it is usually very difficult to check if
a variation is currently ongoing in another MS for a nationally authorised product that addresses
the same safety issue. It would be useful for all assessors involved in signal management to have
access to data of ongoing safety variations at national level.

EMA website for referrals

The EMA website for referrals provides information on ongoing and completed referral proce-
dures for medicinal products. This information can be of added value if the concerned medicinal
product(s) have been assessed through a referral procedure.

The referral assessment report and the recommendations from the EMA can be checked to de-
termine if the new information supporting the signal may change the conclusions made at the
time of finalisation of the referral.

Scientific literature
Review of the scientific literature can give useful information to assessors for the review of a
signal, especially for signals originating from a national database or EudraVigilance. Checking for
any publications regarding similar case reports, pharmacoepidemiological studies or suggestive
of a potential mechanism of action, can provide stronger evidence and warrant a further assess-
ment of the signal. Within SCOPE WP8 a document is developed with recommendations on the
identification of available data sources outside of spontaneous reporting, which the assessor
may find useful.

International and national databases

For signals detected in a spontaneous national database, EudraVigilance and other international
databases, such as VigiBase from the WHO, or FAERS (Adverse Event Reporting System) from
the Food and Drug Administration (FDA), can be a useful data source to cross-check the data. A
further review of the corresponding case narratives may provide further evidence on the potential
association.

In the same way, for signals that are detected from other data sources, such as literature, it is
useful to check national databases and/or EudraVigilance, in order to see if more case reports
can be identified, to assess what further action might be needed.

31
SCOPE Work Package 5
Signal Management – Best Practice Guide

(Pre-) Clinical data

When available, data from preclinical or clinical studies might provide further evidence for the
assessment of a signal. If the clinical trial is large enough and the specific event was investigated,
this usually offers very good evidence for or against the signal.

However, it should be noted that even when an event is not observed in a clinical trial, this should
not be considered as evidence supporting a lack of association, as clinical trials usually are not
able to detect rare adverse events (AEs) or events with a long time to onset.

European studies registry

The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP)
is a network of over 170 research centres, existing networks and providers of healthcare data.
The network is coordinated by the EMA. Its centres conduct independent studies focusing mainly
on the safety of drugs.

The ENCePP database is accessible for free online (19) and can be used to search for ongoing
and finalised studies related to a safety issue and an active substance of interest. Summary of
protocols are also publicly available, as well as interim and final results for studies registered in
the ENCePP database. Therefore, this database can be useful for assessors during the assess-
ment of a signal. It may be useful to check if there is an ongoing study that is relevant for the
signal, which provides useful information within an acceptable timeframe, or if it may be beneficial
to check with the corresponding investigator.

Drug utilisation data

Drug utilisation data on national exposure and patterns of drug use in different subpopulations
can be used to put the case reports into context and for the calculation of reporting rates at
national level.

Exposure data at European or national level can usually be obtained from PSURs. However, for
more comprehensive drug utilisation data, such as information on the number of prescriptions in
a specific country, access to a drug utilisation registry is required. In some MSs such registries
already exists as a readily accessible source of information, while for others, access to this kind
of information is difficult to get.

A comprehensive overview of drug utilisation databases in Europe is made available by the IMI
PROTECT group and is available online (20).

When compiling drug utilisation data from different countries, local legislation, clinical guidelines
and clinical practice should be considered, as they influence drug utilisation patterns.

32
SCOPE Work Package 5
Signal Management – Best Practice Guide

EMA signal validation table

The EMA circulates monthly to NCAs a signal validation table. The table contains a summary of
all signal reviews undertaken by the EMA signal detection team for centrally authorised products
and could, for example, provide additional information for evaluation of the data that generated
the signal.

5.3 Signal detection

According to the IR ‘The competent authority of each Member State shall be responsible for
monitoring the data originating in the territory of that MS’ (IR Art 18(4)).

MAHs2, NCAs and the EMA should continuously monitor the data in the EudraVigilance to deter-
mine whether there are new risks or whether risks have changed. For substances found in na-
tionally authorised products, the monitoring of ADR reports is shared between the NCAs as per
the ‘List of substances and products subject to worksharing for signal management’ (21), and
the NCAs also monitor all medicines for which no LMS has been appointed. The EMA leads on
monitoring EudraVigilance for centrally authorised products.

The legislation does not provide a specific frequency for this monitoring, and states that it should
be ‘with a frequency proportionate to the identified risks, the potential risks and the need for
additional information’. The IR also states that the EMA shall support the monitoring of the Eu-
draVigilance database by providing NCAs with access to data outputs and statistical reports,
customised queries and statistical signal detection methods. In line with the current GVP IX (1),
statistical outputs (eRMRs) are produced from ADR reports received in EudraVigilance every two
weeks for products subject to additional monitoring, and monthly for all other monitored prod-
ucts.

The IR acknowledges that ‘As a general principle, signal detection should follow a recognised
methodology’. However, the methodology may vary depending on the type of medicinal product
it is intended to cover’. The detection of a signal shall be based on a multidisciplinary approach
(IR Art 19(2)).

5.3.1 Signal detection methodology

Best practice
Heterogeneous approaches to signal detection are employed across the NCAs. This
is a strength of the European signal management system and should be fostered,
since this will enable the EU network as a whole to perform high-quality signal detection and
allow for different signals being identified (22).

2 The enhanced access to EudraVigilance for MAHs, to comply with their monitoring obligations, is expected to
come into effect in third quarter 2017.

33
SCOPE Work Package 5
Signal Management – Best Practice Guide

Sources of signal detection

Although spontaneous reporting databases are an important focus for signal detection, a signal
can arise from a wide variety of sources, such as aggregated data from active surveillance sys-
tems or studies, literature information or other data sources.

For example, a signal can arise from a literature finding, and consequently be validated and en-
tered into EPITT. Pontes et al (23) present four examples where a safety signal was detected from
a literature report and had an impact on the benefit risk profile of a drug.

Qualitative and quantitative signal detection

In the scientific community there is agreement that statistical detection methods alone are not
sufficient to detect all signals in spontaneous reporting databases (24-26). Therefore, a combi-
nation of quantitative and qualitative methods is preferable. In some situations there is no need
to implement a quantitative method. If the (MS) ADR database is small, a qualitative review will
be complemented by simple metrics, e.g. the number of case reports is equally useful (5).

Choice of disproportionality methods and selection of adequate detection threshold

Quantitative signal detection relies on statistical disproportionality methods, which are based on
comparison of the observed with the expected count of a drug-event association. It is worthwhile
to reiterate that a signal of disproportionate reporting (4) does not necessarily constitute a signal
of suspect causality. There are currently several quantitative signal detection methods in use,
each with so-called “implementation decisions/rules”. For example:

 The selection of specific thresholds for signals of disproportionality to be raised

 Use of additional criteria, such as number of case reports, for signals of disproportionality to
be raised

 Deciding on the level of precision at which the data mining is performed.

 Decisions on which case reports to perform the detection on (overall or subgrouping of the
database, e.g., vaccines only)

 Deciding whether to look only at suspect drugs or at all drugs regardless of role noted in
ICSRs.

MSs use different methods, including qualitative or rule-based methods only (e.g. only number
of case reports) with varying implementation decisions. Some studies (24, 25) have compared
various disproportionality methods in order to determine which one has the best performance.
Existing evidence indicates that there are no significant differences between Bayesian and fre-
quentist approaches as regards to performance, and that it is possible to achieve comparable
performance with any method.

34
SCOPE Work Package 5
Signal Management – Best Practice Guide

The way methods are implemented for a specific database and the associated decisions seem
to be more important than the method itself. Recent research from the IMI PROTECT group (26)
reinforced this by showing that the choice of signal detection method does not have a substantial
impact on the results and should be rather chosen based on convenience and tailored to the
database. Since thresholds are database-dependent, universally valid thresholds or other imple-
mentation decisions are not possible to recommend. Still, as a guidance, some research showed
that when the count of a specific drug-event combination increased from 3 to 5 this leads to a
decrease of false positive signals when using the EV database (27).

Any signal threshold used should be tailored to the size of the database, the product it contains,
and the level of review that is expected. This applies to complex disproportionality methods, as
well as to rule based approaches (such as review of all fatal case reports). Equally, if changes are
implemented to an already agreed signal threshold, ideally, the new approach should be tested
to ensure that it can both detect safety issues that have been identified in the past for the prod-
ucts under review, and that it is no worse than the previous approach at identifying safety issues
of interest for review in a side-by-side comparison.

In conclusion, changes to the threshold should not be made ad-hoc, and should be tested/ jus-
tified before implementation. If a MS wants to conduct such a change, Table 1 provides guidance
on steps that can be followed.

Table 1. Steps taken before a change in signal detection method or threshold is

implemented

Steps by step process for a change in signal detection methodology or threshold

1. Specify the need/reason for the change (increase in volume of case reports received,
new evidence emerged which recommends an updated process)
2. Develop study plan with relevant experts (e.g. Pharmacoepidemiology Unit, signal
detection group, scientifically trained staff). The plan should include the methods that
will be tested, timelines and testing methods, and any pre-analysis that is necessary to
facilitate the study

3. Approval of the study plan by relevant persons

4. Conduct the study. From previous experience, at least 6 weeks of direct comparison of
the old and new method/threshold is recommended as well as a retrospective analysis
to ensure signals previously identified will not be missed
5. Discuss the results in a meeting with the relevant experts, including assessors involved
in signal management, including positive and negative impacts of making each change
(if multiple changes are tested)
6. If changes are proven successful, next steps would be:
 Agree a proposal for implementation
 Seek feedback from the managers
 Implement changes when agreement is attained

35
SCOPE Work Package 5
Signal Management – Best Practice Guide

The methods and thresholds used at the moment in various national organisations are presented
in Annex 4, with the caveat that no specific method can be recommended in general. Since da-
tabases change over time, a regular justification of the method used (as described above), should
be performed.

MedDRA level at which signal detection should be performed

With regards to the level at which signal detection can be performed, higher levels of the hierar-
chy or Standardised MedDRA Queries (SMQs) do not seem to improve signal detection (28,29)
and might even delay signal detection. Nevertheless, SMQs or other aggregation levels may
sometimes be useful for further signal assessment once a signal is detected.

Stratification and subgrouping

The recently finalised Innovative Medicines Initiative (IMI) PROTECT (15) project offered useful
insights into the potential utility of stratification and subgrouping when performing routine signal
detection. The researchers compared performance of subgroup and stratified (adjusted) analyses
to the unadjusted values within five different spontaneous databases of various size and compo-
sition. Variables included age, gender, time period, country/region of origin, vaccines/drugs and
event seriousness. The conclusions reached were that subgroup analyses consistently per-
formed better than stratified ones, and subgrouping might be good strategy in large, international
databases. The variables with the most impact were age and country of origin.

5.3.2 Monitoring EudraVigilance with a risk-proportionate approach

Best practice
The classification in certain risk categories for monitoring products in EudraVigi-
lance is not standardised, but often MSs base their risk-based approach on 1-2
characteristics of the drug, e.g., time on the market, recent media attention or referrals ongoing
or other regulatory obligations.

An evaluation of the existing frequency of EudraVigilance monitoring and the experience accu-
mulated so far would be helpful in defining a more risk-proportionate consistent frequency of
monitoring to be implemented by MSs.

5.4 Signal validation

According to the IR, signal validation is ‘the process of evaluating the data supporting a previ-
ously detected signal in order to verify that the available documentation contains sufficient evi-
dence demonstrating the existence of a new potentially causal association or a new aspect of a
known association, and therefore justifies further analysis of the signal’ (IR Art 21)).

36
SCOPE Work Package 5
Signal Management – Best Practice Guide

As already explained (see section 5.2), during the validation of a signal, the evaluation of the data
should be aimed at deciding if further analysis is necessary and the focus should be on deter-
mining if the signal reflects new information and if it is at least a reasonable possibility. For signals
originating from spontaneous reports, that would include, as a minimum, making sure that the
signal is not only based on duplicate reports and that there is a plausible time to onset.

Best practice
Depending on the organisation and the resources at national level, validation of the
signal might differ between MSs. Some MSs have an (optional) additional decision-
making loop at national level or other standard operating procedures in place. A more
extensive evaluation of data might already take place before next steps are taken.

A relatively detailed guidance on aspects that should be considered when evaluating the evi-
dence supporting a signal, is outlined in GVP Module IX (5).

In order to validate, confirm or assess a signal, the previous awareness of the possible associa-
tion, the data supporting the signal and the availability of additional data from other relevant
sources of information should be taken into account.

For assessors involved in signal management, a practical and hands-on checklist that summa-
rises the most important elements, points to consider and relevant sources of information when
evaluating the data supporting a signal would be beneficial.

Example: signal validation and assessment checklist

An example of a signal validation and assessment checklist has been developed
within this BPG and is outlined in Annex 2. The checklist covers elements from signal
detection, validation and assessment. Depending on the step in the signal management
process, and to what extent the validation of the signal takes place, the checklist can be used
with different aims by assessors evaluating the data, to help ensure that all relevant sources of
information are considered and the most important actions are performed. This checklist is meant
to serve as an extra helping tool for assessors during the validation, confirmation or assessment
of signals or ICSRs. It should not be considered mandatory or totally exhaustive, as some other
important considerations might have not been included and not all points have to be considered
during every step of the signal management process.

37
SCOPE Work Package 5
Signal Management – Best Practice Guide

5.4.1 When to involve Marketing Authorisation Holders in early signal

management
The EU signal management process concerns all stakeholders involved in the safety monitoring
of medicinal products, including patients, healthcare professionals, MAHs, regulatory authorities,
scientific committees and other relevant decision-making bodies. Each one of the stakeholders
has specific roles and responsibilities in the signal management process. The MAH mainly has
the responsibility to perform the continuous monitoring of the safety of its medicinal products
and inform the authorities of any changes that may have an impact on the benefit/risk profile for
the medicinal product.

Best Practice
For national safety issues, at the first review step, or whenever additional information
is needed, MAHs could be contacted by MSs at their own discretion. Equally it may
be appropriate for a MS to approach a MAH where they believe they may have identified a safety
issue, but do not yet have enough information to validate it as a signal. However, once a signal
has been entered into EPITT, the EU signal management process should be followed.

The EMA Questions and Answer document on signal management (6) clarifies how MAHs are
informed that a signal will be discussed at the PRAC, and how they are informed of the outcomes
of the PRAC signal assessment.

5.4.2 Regulatory confirmation of validated signals

Signal confirmation is a formal step in the signal management process according to the IR. Con-
firmation by the PRAC Rapporteur or the LMS means communicating through EPITT that the
signal requires further analysis.

According to the IR, a validated signal that requires further analysis shall be confirmed as soon
as possible and no later than 30 days from its receipt (4).

A MS shall confirm or non-confirm signals on active substances for which it has been appointed
the LMS or PRAC Rapporteur. For active substances with no signal lead appointed, the MS shall
confirm any signals detected and validated in its territory (30).

A validated signal is shared within the EU-network through EPITT for subsequent action. This can
either be confirmed (meaning discussion at PRAC for further assessment and decision-making)
or not confirmed (no discussion at PRAC). The EU signal lead, the MS itself, or the Rapporteur
will decide whether there is reason to confirm or not confirm the signal (IR Art 21(3)). If the signal
is marked as confirmed, it is brought to the PRAC agenda for discussion to determine the best
course of action. If the signal is marked as non-confirmed, it is not brought to the PRAC agenda.
Any signal can be raised again at a later stage if new evidence arises which supports it.

38
SCOPE Work Package 5
Signal Management – Best Practice Guide

Best Practice
In the EU signal management process, signal confirmation could be described as a
second opinion on the initial results of signal validation at the NCA level and it
should be differentiated from signal validation. Signal confirmation is not a full assessment of the
signal, and a confirmed signal does not necessarily mean that a causal relationship has been
demonstrated. Confirmation in EPITT only means that the signal will be brought to PRAC for
discussion at the EU level and further decision-making. The decision, i.e. actions to be taken at
the EU level, is up to the PRAC.

It is important to keep in mind that signal confirmation as outlined in IR only relates to a regulatory
procedural step in EPITT.

5.5 Signal prioritisation

Signals with a potential important public health impact, or which may significantly affect the ben-
efit-risk profile of the medicinal product, require urgent attention and should be prioritised for
further management.

Best practice
Different factors may be taken into account for the prioritisation of signals (IR Art
20(2)) (4), namely whether:

 The association or the active substance/medicinal product is new

 The strength of the association

 The seriousness of the reaction involved

 The documentation of the reports in the EudraVigilance database

Before the IR came into effect, the CIOMS VIII guidance (2) proposed a two-step signal prioriti-
sation process, based on a combination of a first impact analysis and a further signal prioritisa-
tion.

Some of the CIOMS VIII criteria are:

 Seriousness

 Rapidly increasing disproportionality score for the impact analysis

 Occurrence during the first few years post launch (newer drugs)

Prioritisation of signals is critical to ensure that acceptable timelines and appropriate resource
allocation will meet public health and regulatory obligations (31). A prioritisation tool can help to
reduce the subjectivity of the prioritisation process and might increase standardisation.

39
SCOPE Work Package 5
Signal Management – Best Practice Guide

A novel tool for prioritising pharmacovigilance issues within the MHRA was developed and im-
plemented, called the Regulatory Pharmacovigilance Prioritisation System (RPPS). The RPPS
tool (see Annex 5) provides a systematic approach to prioritise signals according to four catego-
ries: health consequences, strength of evidence, regulatory obligations and public perceptions.

Other tools are described in the literature, such as the 2012 FDA (32) publication of a draft guid-
ance for Classifying Significant Post-marketing Drug Safety Issues. Another prioritisation tool was
developed and tested in a MAH database by Levitan et al (33).

5.6 Signal assessment

According to the GVP Module IX the objective of signal assessment is to further evaluate a vali-
dated signal so as to identify the need for additional data collection or for any regulatory action.
The preceding signal validation is essential to evaluate whether there is a new potentially causal
association, or a new aspect of a known association, that justifies further analysis of the signal.
The signal assessment should take into account all available data in order to further increase the
strength of the evidence to reach a high-quality decision and signal outcome. Following the PRAC
analysis and prioritisation, the signal assessment is performed by the PRAC Rapporteur or LMS
and finalised at PRAC.

The aim of the signal assessment step should be to decide whether a regulatory action is neces-
sary, and the focus should be on reaching a final decision on the causal relationship and to con-
sider the need for (additional) risk-minimisation measures.

The data reviewed at this step is usually more extensive. Different sources of information are
available to MSs for signal assessment, of which the most frequently used are published litera-
ture, expert consultation and additional data provided by MAHs.

Best practice
The IR (4) gives only high-level instructions on how to perform assessment. All the
available pharmacological, non-clinical and clinical data and information from other
sources should be reviewed, and the strengths and limitations of each source should be carefully
considered.

Sources of information mentioned in GVP Module IX (5) are:

 The application dossier

 Literature articles

 Spontaneous reports

 Expert consultation

 Information held by MAHs and competent authorities

40
SCOPE Work Package 5
Signal Management – Best Practice Guide

When PRAC recommends further assessment of a signal, PRAC will nominate a Rapporteur for
the assessment of the confirmed signal with a timeframe for the assessment. Sometimes a further
assessment is not required within the signal management process, if a thorough assessment has
been performed in previous steps, or if the safety issue is dealt with in another procedure (e.g. in
a PSUR or variation).

While comprehensive scientific literature is already available on signal detection, MSs must
mainly rely on overall guidance when it comes to signal assessment. Little guidance can be found
in the scientific literature, and the EU network is encouraged to facilitate collaborative research
across the network and academia. The outcome of such research and sharing of knowledge
could also contribute to better training of signal assessors in the future.

Example
An example of a signal validation and assessment checklist is outlined in Annex 2.
The checklist covers elements from both signal validation and assessment. It can be
used by signal assessors to help ensure that relevant sources of information are consid-
ered and the most important actions are performed in order to deliver consistent and high-quality
assessments.

Signal assessment is a challenging discipline and MSs are expected to deliver high-quality as-
sessment to the EU-network. A thorough and high-quality signal assessment is the base for any
regulatory decision in signal management, and any decision should be based on the most com-
prehensive level of evidence as possible.

Therefore, insight is important into what level of evidence is needed in the assessment, to trigger
regulatory actions for drugs and vaccines with different risk profiles. To suggest answers to ques-
tions like how many case reports are enough to trigger a signal and when to update the SmPC,
there is a need to bring together and share knowledge across the EU-network.

Example
A paper from 2015 by Segec et al about Strategy in Regulatory Decision-Making for
Management of Progressive Multifocal Leukoencephalopathy (34) is a good example
of high-level knowledge sharing within the EU network. A tool for regulatory decision
management of PML was developed and piloted, and the authors found that the methodology
developed for PML was a rational approach to consider available evidence and could improve
regulatory decision making. Others are encouraged to use and modify this methodology on other
topics.

41
SCOPE Work Package 5
Signal Management – Best Practice Guide

5.7 Reports of special interest

ADR reports frequently relate to areas of special interest, either scientifically, politically, or from
a regulatory perspective. They may relate to events occurring in a particular population, for in-
stance paediatrics, geriatrics or use in pregnancy, or could relate to products with unique risks
or considerations, such as biologicals or vaccines. Alternatively, reports could highlight concerns
about how the product has been used, either by the healthcare professional (medication error,
occupational exposure), or by the patient (medication error, misuse, abuse). Such scenarios may
warrant additional attention within the signal management process, due to the potential safety
risks.

5.7.1 Population-based approaches

Best practice
Spontaneous ADR reporting systems span across a broad range of populations.
This diversity can result in differing safety profiles and should not be ignored. The
pharmacodynamics and pharmacokinetics of a medicine may be very different in children com-
pared with adults (35) in relation to the way they metabolise and excrete medicines and can have
harmful consequences. The same can be said for geriatric populations (36). In pregnancy, reports
need to be considered in context of both the parent and the child. Having additional strategies in
place to cover the diversity and potential confounding factors could result in fewer false signals
being flagged, fewer missed signals and less masking (37).

Recent research from the IMI PROTECT group (15) recommended when statistical analysis is
applied that subgroup analysis may be beneficial in routine first pass signal detection and should
be considered and stratified/adjusted analyses are unlikely to provide added value. Subgrouping
by population calculates disproportionality statistics within each individual stratum separately,
aiding higher quality signal assessment by increasing sensitivity and precision in these groups.
As a result of this research, the EMA are implementing the use of subgroups for paediatrics and
geriatrics into the eRMRs after a successful pilot phase, which showed them to be beneficial.

As an alternative to the review of this additional data, as part of the routine signal detection pro-
cess it may be appropriate to define strategies for review of special population groups at a re-
duced frequency compared to routine evaluation. Dependent on the size of the overall database,
and the volume of case reports in the analysis period, this approach may offer the advantage of
providing a larger volume of case reports making the regulatory decision-making process easier.
However, case reports should not be excluded from routine signal detection to ensure timely
identification of the most serious issues. MedDRA have published modifiable paediatric and gen-
der alert term lists that could be used to support pharmacovigilance activities, especially when
age/gender information is not available (38).

42
SCOPE Work Package 5
Signal Management – Best Practice Guide

5.7.2 Product/substance based approaches

Best practice
Vaccine reports were considered a priority for most MSs, however awareness of
other product- or substance-based methods was very low.

There is detailed guidance on vaccine signal detection processes in the GVP module on ‘Product
or Population Specific Considerations I: Vaccines for prophylaxis against infectious diseases’,
which should be followed where appropriate. A signal may also relate to evidence of reduced
efficacy or effectiveness, vaccine failures and quality deviations with potential impact on safety,
efficacy or effectiveness and could be batch specific. Therefore, signal detection for vaccines
should be considered at both brand and batch level.

Evidence from PROTECT found that during statistical analysis ‘subgrouping by vaccines/non
vaccines resulted in a decrease in both precision and sensitivity in all spontaneous databases.
This was almost exclusively driven by the vaccines subgroup.’ These effects were due to the
suppression of known vaccine ADRs as a result of comparing vaccines to each other which alt-
hough is desirable for certain reactions such as injection site reactions, it is not ideal for more
serious reactions (e.g. Guillain–Barré Syndrome). PROTECT therefore recommend that careful
consideration should be applied before taking this approach. This emphasises the importance of
assessors understanding the rationale underlying selection of statistics and thresholds that have
been applied to the data to ensure that the information presented is used appropriately.

One potential way to lessen the effect of known ADR suppression is to apply rule-based criteria
to signal detection to identify and always detect certain reaction terms of interest that, under the
above circumstances, would be masked. Multiple MSs are using a list of predefined important
terms, which flag up reports for assessment. This is explained further in the next section.

There is some research evidence (39) that suggests that, along with new and first in class drugs,
more signals are identified for biologicals, highlighting the importance of strengthening analysis
of them. Again, full guidance is available in the second product or population-specific consider-
ation GVP module for biological medicinal products (17). However, broadly, the principles de-
scribed above for vaccine signal detection are common for other biologicals as well.

43
SCOPE Work Package 5
Signal Management – Best Practice Guide

5.7.3 Reaction-based approaches

Best practice
When applying reaction-based approaches, most MSs focus on serious medical
events and terms frequently associated with ADRs. Various stakeholders have indi-
vidually developed lists of specific drug-related events; taking into account the seriousness of
the event and its likelihood of being drug induced. These event terms are prioritised during signal
detection, as they could more likely be true signals or potentially have a larger public health im-
pact. These strategies can help ensure that important reactions terms are highlighted amongst
other terms for assessment. Some examples of these lists include:

 Designated Medical Events (DME): those AEs which are rare, serious or which are more likely
to be associated with a high drug-attributable risk, for example Stevens-Johnson syndrome
(40)

 Targeted Medical Events (TME): other events of special interest associated with particular
medicinal products and/or patient populations

 Important Medical Events (IME): a selected group of events that should be considered seri-
ous. This list was developed by MedDRA MSSO organisation and coordinated by the EMA
(41).

A reaction-based approach can look at these terms regardless of disproportionality or in combi-

nation with disproportionality analyses when the numbers of ADR reports are large. They have
been implemented into the prioritisation of drug-event combinations in eRMRs to help define the
criteria and number of drug-event combination rows to review.

Example
Guillain–Barré Syndrome is a serious medical event thought to occur due to effects
on the immune system, and occasionally reported after the administration of a vac-
cine. It is a known event for many vaccines, although causality is difficult to establish, and
is commonly reported across all vaccines. As a result, when subgrouping by vaccine/non-vac-
cine, an extremely large number of case reports would be required for the disproportionality score
to be raised. Studies performed in IMI PROTECT (31) indicated that this event was supressed
within a vaccine subgroup. One MS described a reaction-based approach to counteract this
problem. They created a list of medical events of interest (including Guillain–Barré Syndrome),
which are flagged for signal assessment regardless of the disproportionality score.

44
SCOPE Work Package 5
Signal Management – Best Practice Guide

MSs should consider using a pre-defined event term list such as the DME list to help prioritise
ADRs for review either regardless of disproportionality, to ensure important events are not
missed, or in combination with disproportionality, to help focus attention on important terms for
selected categories.

5.7.4 Other specified groups of interest

Best practice
The definition of ADR in the EU legislation (4) has been expanded to include reports
of noxious and unintended effects resulting from medication errors, misuse, abuse
and occupational exposure and from uses outside the terms of the marketing authorisation (off-
label use). These are often grouped together into a cluster.

It is important these are identified, so that their risk can be minimised, unlike other signals they
do not require the establishment of a causal relationship because they are unintended events
caused by human error.

No robust statistical methods for automated identification of ADRs from this group are identified,
and further research and/or development is recommended in this area to enable efficient identi-
fication of these safety issues, which are well understood to have a significant impact on the
healthcare system and are often easily preventable. Some of the barriers to analysis of this area
are either a lack of data, or an incomplete dataset. Specifically, different national bodies, separate
from the NCAs for pharmacovigilance issues, but part of, or linked to, the healthcare system, are
often responsible for collection and analysis of such data. Even where data sharing arrangements
are in place, these organisations have different goals and purposes, and will not necessarily hold
the data in a format easy to analyse alongside NCAs’ ICSR data specifically for pharmacovigi-
lance and signal management purposes.

Detection and management of medication errors, misuse and abuse and occupational exposure
is unlikely to be the sole responsibility of the NCAs and, as such, it is important to establish or
improve relationships between the NCA and relevant areas of the healthcare system. Such rela-
tionships should enable sharing of data both to help a MS carry out its pharmacovigilance re-
sponsibilities, but also to enable the healthcare network to reduce and learn from incidents that
have occurred. Such collaborative networks may also be valuable when sharing other outputs of
the signal management system, such as communicating relevant risk-minimisation measures.

45
SCOPE Work Package 5
Signal Management – Best Practice Guide

Example
A signal of medication errors potentially leading to inappropriate dose administration
was identified by a MS following a communication from their national Organisation for
Medication Incidents (42). They reported cases of 8 patients who received a higher dose
than intended, including 2 fatalities. This led to a review of their national case reports, which
subsequently led to the signal being raised, resulting in preventative measure being put into
place. This example highlights the value of NCAs working alongside other areas of the healthcare
system.

The working model guidance for exchanging information between different organisations pre-
sented in this medication error guidance could also be an example for working with other organ-
isations, for example national poison centres.

There is detailed guidance on the recording, coding, reporting and assessment of medication
errors available in the Good practice guidance on recording, coding, reporting and assessment
of medication errors (43), which was published by the EMA in 2015. It is important to note that
medication error terms should only be recorded against a case when an error has been stated
by the original reporter. No assumption should be made by the MS. However, it is worth noting
that, in line with Good practice guidance on medication error (43), if a MS (or MAH) thinks a
medication error code is warranted, an alternative classification can be provided in the ICH E2B
(R2) data element B.5.3 ‘Sender’s diagnosis/syndrome and/or reclassification of reaction/event’.
This information could also be used as a means of identifying reports concerning medication
errors by creating queries in EudraVigilance to search for relevant MedDRA terms in both the
‘Reaction/event’ and ‘Sender’s diagnosis/syndrome and/or reclassification of reaction/event’
data elements, resulting in a list of suspected products associated with medication errors.

Because of the nature of the way this data is obtained and coded it may be challenging to analyse
using traditional statistical methods used in Pharmacovigilance. The MedDRA MSSO imple-
mented a 27th System Organ Class (SOC) in MedDRA, which contains product quality issues.
Some of these areas may provide secondary linkage to medication errors, however, the most
valuable asset in identification of these case reports if they have been coded is likely to be the
Medication Errors SMQ. IMI PROTECT did not find improved detection performance when com-
paring SMQs to PTs (29) generally, but this does not exclude their use for specific circumstances,
such as medication errors. Should such approaches be adopted it will be important to ensure
consistent coding practices to identify relevant case reports, or that routine methods are capable
of detection of the most serious safety issues.

Because of the likelihood of missing signals from this group due to the issues described above
it is important that MSs signal management system consider the potential for medication errors
within routine activities, and that any other approaches should be considered as additional
methodologies.

46
SCOPE Work Package 5
Signal Management – Best Practice Guide

Example
Leuprorelin is indicated for the treatment of palliative advanced hormone-dependent
prostate cancer and works by suppressing testicular steroid genesis. A signal was
detected by one MS following a report from a healthcare professional concerning 3 re-
ported cases of increased testosterone levels after treatment with Leuprorelin (44), suggesting a
lack of efficacy. It was noted in the reporter’s comments section of the report that it could be due
to ‘a wrong technique in drug administration process’. A further search of the MS’s database
found 11 similar reports, which suggested either a lack of efficacy and/or a wrong technique
being used; however, only 4 of these had an adverse reaction belonging to the medication error
HLGT. This signal emphasises that the coding of medication errors cannot be relied upon alone
and that qualitative analysis is beneficial. Medication errors need to be considered when looking
at all signal outputs.

5.8 Data quality

Best practice
Good data quality is relevant for all steps of the signal management process: signal
detection, prioritisation, validation and assessment. In addition to incorrect coding,
ICSR data quality aspects are to be considered – for example, duplication of reports, absence of
information in structured E2B fields, and lack of completeness of ICSR. Recent studies (45)
showed that more informative reports (with a minimum of required fields completed) are
correlated with true signals. Data quality considerations are addressed on a high level in the EU
legislation and more detailed guidance is provided in several of the GVP Modules. In general
terms, reference is made to quality management systems, procedures to collect follow-up infor-
mation and to the relevant standards and terminologies that should be applied by all stakehold-
ers. The EMA has procedures in place to manage duplicate ICSRs and to review the quality of
the ICSRs for all organisations reporting to the EudraVigilance database (30).

Consistent and standardised coding of data is important and highlights the importance of appro-
priate training in the use of MedDRA terminology, especially at the time of data entry, to improve
the data quality. With electronic E2B reporting, ICSRs may be automatically loaded to the data-
base without manual intervention. As a result, modern EU spontaneous reporting databases con-
tain data collected by various organisations, all with their own methods of data collection, data
processing, coding, etc. As incorrect or inconsistent coding have an impact on data retrieval and
analysis processes, SCOPE WP5 reiterates the importance of following the existing guidance on
data quality and coding.

47
SCOPE Work Package 5
Signal Management – Best Practice Guide

Specific skills and knowledge, including clinical knowledge, at the level of the coding staff are
required to perform coding (46). Training should be provided to all MS staff involved in ICSR
coding in order to achieve the optimum level of coding and to ensure that all the parameters for
quality are achieved. The training in MedDRA dictionary of personnel responsible for signal de-
tection activities, is also important. Online training webinars for both coding and information re-
trieval are available on the MSSO website; for further information on coding of AEs, please refer
to the ‘MedDRA Term Selection: Points to Consider’ and ‘MedDRA Data retrieval: Points to Con-
sider’ documents (47,48).

Coding is also highlighted as key challenge for reports of special interest. It is unlikely that, for
instance, reporters completing online web forms will select accurate MedDRA PTs related to
special situations, when offered the possibility, but rather capture this information in free text.
Clearly this creates challenges in the analysis of the data, some of which are described above.
The solution to these problems is not considered to be as simple as the addition of extra ques-
tions to the reporting systems, and users may have different understandings of terms to those
used in the regulatory setting. For example, addition of questions around whether a medication
error has occurred have been found to be answered in different ways depending on the stake-
holder (based on MHRA implementation experience, unpublished). Patients frequently selected
the medication error check box, simply because they had experienced an ADR and therefore
considered the medication had been prescribed to them in error (based on MHRA implementa-
tion experience, unpublished). Although changing the wording of the question and guidance can
improve this information, it is important to consider that the reporter’s perspective may differ
depending on their individual circumstances. Guidance from SCOPE WP4 should be used to
develop harmonisation in this area across the EU network to aid consistency within signal detec-
tion, including reports of special interest.

As a prerequisite for signal management, the importance of data quality is emphasised and all
stakeholders should devote appropriate attention to improve it.

5.9 Training
Achieving the required quality for the conduct of pharmacovigilance processes and their out-
comes by an organisation is intrinsically linked with the availability of a sufficient number of com-
petent and appropriately qualified and trained personnel and having a specific recruitment plan
to ensure the competences and skills would be of value (49). As laid down in the legislation, all
personnel involved in the performance of pharmacovigilance activities shall receive initial and
continued training (IR Art 14(2)).

48
SCOPE Work Package 5
Signal Management – Best Practice Guide

Best practice
It is recommended that MSs have a training plan in place to provide staff members
involved in signal management with the knowledge, guidance and methods to
perform signal management activities in line with their job descriptions. The extent of
the specific topics covered should fit the role, experience, knowledge and expertise of the staff
member. Such training plan could cover:

 Relevant internal Standard Operating Procedures and Working Instructions

 Guidance regarding signal detection methods: Staff members should be made aware of the
signal detection method(s) applied in their organisation and also on how to use the eRMRs
from EudraVigilance

 Background documents: Staff members should be made aware of the most relevant guide-
lines for signal management (e.g. Report of CIOMS Working Group VIII, EMA guideline on the
use of statistical signal detection methods, EU SmPC guideline, GVP Module IX on Signal
Management and the EMA Questions and Answers document on signal management). The
most updated version of the documents should always apply

 EPITT: It is recommended to attend the EPITT webinar training provided by EMA

 MedDRA coding and analysis: The MedDRA MSSO provides a number of different training op-
portunities, including free face-to-face trainings for regulatory authorities. An online free intro-
ductory training curriculum covers MedDRA coding, as well as data analysis (including SMQs)

 EudraVigilance Data Analysis System (EVDAS) training for NCAs organised by the EMA

 Literature search: The staff member should be trained on how to perform literature searches
and set up alerts in order to support signal management activities

 Relevant regulatory documents: Staff members should be introduced to the most relevant
documents used for signal detection and management, such as SmPCs, PSURs, RMPs, re-
newals, referrals, variations, as well as the corresponding Assessment Reports. This should
also address how to access the documents and information

 External training courses in therapeutic areas and in other topics of interest: In view of the
need to maintain and expand the knowledge regarding pertinent topics (therapeutic areas,
epidemiology, statistics, pharmacovigilance, etc.), it is encouraged to attend external
courses, conferences and congresses in order to build expertise and keep up to date with
new developments. The EU network training centre (EUNTC) can be consulted to check rel-
evant training opportunities

This SCOPE WP5 guidance document will be supported by training provided in the context of
the SCOPE project. This training will focus on the signal management process and an e-learning
module will be made available for use after the SCOPE project has finished.

49
SCOPE Work Package 5
Signal Management – Best Practice Guide

6. Conclusions

The SCOPE WP5 team identified a number of challenges that MSs are experiencing with the
interpretation, and some complexities in implementation, of the existing signal management leg-
islation. SCOPE WP5 identified challenging areas and best practice through a questionnaire; of
particular importance to the MSs was clarification of requirements regarding the prioritisation,
validation and confirmation of signals, as well as the processes for reports of special interest.

In response to these findings, the SCOPE WP5 team created this document of recommendations
and BPG to support the legislation and GVP on signal management and make the principles
within these documents more widely achievable for all MSs, regardless of the size or structure of
the NCAs.

Signal detection methods in individual MSs are not and do not need to be the same, and indeed
there is strength in their diversity. Depending on the way the signal management is organised in
the different MSs, signals that are brought to PRAC may differ in maturity. Some MSs have ad-
ditional decision-making loops in place, where some aspects of the signal assessment come into
play, whilst others do not. Access to all the necessary documents and data sources is crucial,
for thorough assessment of signals and measures should be taken to ensure that they are avail-
able to all assessors involved in signal management, where this is not already the case.

In addition to steps that can be adopted at a national level, there are a number of recommenda-
tions for the EMA and the EU Regulatory Network to consider as a whole. These cover both
processes and IT tools used within signal management and demonstrate the possibility for further
efficiencies within the system. Please see Annex 1.

The BPG gives clear and specific advice as to how these recommendations can be achieved,
with an aim to improve the quality of signal management throughout Europe. While some can be
implemented by individual MSs, others are specific to the EMA or require collaboration across
the European Regulatory Network.

50
SCOPE Work Package 5
Signal Management – Best Practice Guide

7. Annexes

Annex 1. SCOPE WP5 recommendations for future research

and development
Process-related recommendations
These recommendations impact the processes and activities of signal management, regardless
of their implementation within the system.

1. A risk proportionate approach (as a function of product type or type of safety issue) to
EudraVigilance monitoring should be developed. By taking the conservative approach and
monitoring all the substances in the same way, resources are diverted and important signals
might be delayed as a consequence. A risk-proportionate approach would allow a better al-
location of available pharmacovigilance resources.

2. The EU network should consider how to facilitate collaborative research in signal as-
sessment methods as a basis for regulatory decision making. Limited scientific literature
is published on signal assessment, and the EU network is encouraged to facilitate collabora-
tive research across the network and academia. The outcomes of such research and sharing
of knowledge could also contribute to better training of signal assessors in the future. Both
the regulatory and academic circles would benefit from an open scientific dialogue and more
research in this area.

IT system-related recommendations
These recommendations impact the IT system (software, network resources and services) re-
quired for the existence and functioning of a signal management system.

1. The EU network would benefit from an integrated EU signal management tool combining
signal detection and tracking functionalities and replacing the existing Excel based com-
ponent. The current eRMR system, although it facilitates the monitoring of EudraVigilance
data, would benefit from further improvements in order to allow access to the EudraVigilance
data to the level of detail desired by MSs. MSs consider that there would be significant value
in a tool that integrates the functionality of the eRMR, EVDAS and EPITT to enable a seamless
and linked signal management approach.

2. The need of a suitable tool to exchange information on preliminary signals within the EU
is identified and the EU network would benefit from sharing signals at an early stage.
This tool would facilitate exchange of relevant information at an early stage, to prevent dupli-
cation of work.

51
SCOPE Work Package 5
Signal Management – Best Practice Guide

3. Systems which allow cross-procedural tracking of final outcomes of signals is identified

as a need (in case the signal is addressed in a different procedure). The outcomes of
signals addressed in other procedures (e.g., PSURS, renewals) are not captured in the current
EU signal management tracking systems and this does not facilitate transparency and effi-
ciency, as it might lead to duplication of work and waste of resources.

4. A structured SmPC database would be helpful to facilitate validation of signals. Checking

if the ADR is reflected in the SmPC is usually the first step in signal validation. Therefore, the
pharmacovigilance system would benefit from an automated tool for this purpose. This tool
is already available for centrally authorised products (50), but not for nationally approved
products. For nationally approved products, a Core Safety Profile (CSP) approach could be
used: a dataset containing the minimum safety information present in the SmPCs from all
MSs.

Recommendations for future research

1. There is a trade-off between sensitivity and specificity with all current statistical detec-
tion methods, and further research is recommended to optimise these approaches in
both EudraVigilance and national systems. The trade-off between sensitivity and specificity
always depends on the subjective value assigned to one true signal lost versus resources
spent on one false positive. Traditionally, in pharmacovigilance a higher premium is put on
sensitivity. The balance however should be carefully selected and justified.

2. Dedicated methodologies for analysis of reports of special interest should be developed.

Limited research was found covering methodologies specific to reports of special interest,
despite their scientific and political importance. The EMA and MSs should work together to
enable such research to be carried out for the benefit of the network.

Further detail on suggestions for further development as identified during

the project

The need for an integrated signal management system at European level

EMA has undertaken significant work in strengthening the statistical methodology included in the
eRMR by implementation of IMI PROTECT recommendations. Furthermore, some improvements
in the layout of the eRMR have been made, in order to enhance user-friendliness and increase
efficiency. However, the fact that the tool remains Excel-based is considered a barrier to further
development of the signal management process. Users feel that the lack of true integration/in-
teractivity between the eRMR and the EudraVigilance Data Analysis System poses a limiting fac-
tor in improvement of the system. This lack of connection between the different tools reduces
efficiency of use of the eRMR and the capacity to assess high volumes of multi-factorial issues
fast. The lack of flexibility for an individual assessor to define layout preferences (as might be
expected in a web-based interface) was also identified as a barrier.

52
SCOPE Work Package 5
Signal Management – Best Practice Guide

There would be significant value in a toolset that integrates the functionality of the eRMR, Eudra-
Vigilance and EPITT together, to enable an efficient and integrated signal management process
across the EU-network. The tools should facilitate users to extract information at whichever point
they need, and view it alongside other information in the format they require. A second function
and the advantage of having such an integrated system would be to allow an audit trail of the
assessment process while simultaneously recording useful information for assessors to consider
in future assessments of the issue. The tools should enable information to be shared without the
need for additional emails, and consideration should also be given to the ability to collaborate on
issues which do not yet meet the legal definition of a signal.

The need for an early exchange of information to inform the EU network and prevent
duplication of work
The EU network would benefit from the possibility of an early exchange of information on signals
in order to inform other MSs and prevent duplication of work. This would provide transparency
of signal detection outputs to other MSs prior to the point that they are confirmed.

Given there are no other tools currently available for this specific purpose, these are the possi-
bilities to facilitate this exchange of information: inform the LMS by email, distribute a NUI, or
enter the signal in European Pharmacovigilance Issues Tracking Tool and propose to the LMS to
not confirm this signal (to ensure that the signal will not be at the PRAC agenda). In the future,
this sharing should be facilitated by the creation of a new signal management tool.

A structured SmPC database to facilitate validation of signals

Checking if the ADR is reflected in the SmPC is usually the first step in signal validation. There-
fore, the pharmacovigilance system would benefit from an automated tool for this purpose. This
tool is already available for centrally authorised products (50), but not for nationally authorised
products. For nationally authorised products, a Core Safety Profile (CSP) approach could be
used: a structured database containing the minimum safety information present in the SmPCs of
all the MSs could be a helpful tool. The CSP approach has been used in the past at the time of
the previous work-sharing where the MAH developed a Core Safety Profile. It would be beneficial
for this dataset to include common information from sections 4.3 – 4.9 present in all SmPCs
within the EU and any relevant safety information from section 4.2.

Systems which allow cross-procedural tracking of final outcomes of signals is

identified as a need (in case the signal is addressed in a different procedure)
The outcomes of signals addressed in other procedures (e.g., PSURS, renewals) are not captured
in the current EU signal management tracking systems and this does not facilitate transparency
and efficiency, as it might lead to duplication of work and waste of resources. It would be bene-
ficial for enhanced signal management tools to be developed that enable aggregation of safety
issues associated with a product, irrespective of the procedure from which they are derived.

53
SCOPE Work Package 5
Signal Management – Best Practice Guide

Tools provided should facilitate a risk-proportionate approach to signal detection

Whilst we cannot make a recommendation from the data attained from WP5 about the most
appropriate signal management strategies for products/substances of differing maturity, it is
clear that different approaches may be beneficial. Any enhancement or redevelopment of the
signal management toolset should take account of the need for flexibility in strategies. This could
include differing review periods for varying maturity, or different thresholds for different groups
of product, cases or interest areas. Tools should offer the ability to rapidly adopt new ap-
proaches, and to study different methodologies to understand their impact. Whilst different ap-
proaches may be adopted it is important that assessors have access to regularly updated dis-
proportionality criteria to enable rapid assessment of a new issue irrespective of the pre-defined
review period.

54
SCOPE Work Package 5
Signal Management – Best Practice Guide

Annex 2. Signal validation and assessment checklist

Validation of the signal
Important sources: SmPC, EPITT, PSUR, RMP, EMA website, other regulatory procedures

1. Is the event reflected in the SmPC of the active substance?

i. For centrally authorised products, check the SmPC available at the EMA website; for na-
tionally authorised products, check the SmPC(s) available in your MS
ii. For active substances for which a generic version exists, check first the SmPC of the
innovator product (if possible)

 Check sections 4.3, 4.4, 4.5, 4.6, 4.8, 4.9 of the SmPC for information regarding the event

 Check if the event might be covered by a similar term or a higher level term in the SmPC

 Check if the event is reflected in the SmPC of another medicinal product containing the
same active substance

 Check if the event is reflected in the SmPC of a medicinal product from the same class

 In case of an interaction, you may check if there is information on the interaction in the
SmPCs of the other medicinal products concerned

2. Does the event reflect a new aspect of a known association?

 Increase in frequency of occurrence

 Change in duration and/or time to onset

 Change in severity

 Change in occurrence pattern (e.g. affecting a specific population)

 Change in previously reported outcome (e.g. new fatal cases)

3. Has the association previously been addressed in a regulatory procedure?

 Check if there is information in EPITT

 Check the PSUR of the medicinal product

 Check the RMP of the medicinal product

 EMA website of referrals

 Recently submitted/assessed variations in your MS regarding same/similar safety issue

Of note: exclude that the signal is only due to duplicates and that there is a plausible time to
onset.

55
SCOPE Work Package 5
Signal Management – Best Practice Guide

Assessment of the data (points to consider)

Important sources: ICSRs, national databases, EudraVigilance, literature

1. Review of the cases supporting the signal

 Number of cases supporting the association (after exclusion of duplicates and cases with
no supporting temporal association)

 Number of cases appropriately documented with sufficient information about, e.g., sus-
pect drug, event reported, demographics (age and gender), indication, outcome, concom-
itant medication

 Consistency of the evidence across cases (e.g. patterns)

 Route(s) of administration and product(s) formulation

 Cluster of reports, e.g., many reports from the same reporter, publication, etc.

 Cases fulfil the diagnostic criteria for the event

2. Strengths

 Biological and pharmacological plausibility (possible mechanism)

 Dose relationship

 Number of cases with positive de-challenge

 Number of cases with positive re-challenge

 Low background incidence of the event

 Time to onset

3. Clinical relevance

 Seriousness/severity of the event

 Reversibility of the event

 Event affecting special populations (e.g. pregnant women, children, elderly) or patients
with pre-existing risk factors

 Events occurring in different patterns of use (e.g. off-label, overdose, misuse, medications
errors)

 Association likely to apply to other active substances of the same class

 Potential for prevention

56
SCOPE Work Package 5
Signal Management – Best Practice Guide

4. Other aspects to be considered

 Possible class effect

 Possible drug-drug interaction

 Possible medication error

 Possible quality issue

 Possible off-label use

 Possible overuse, abuse, misuse

5. Weaknesses

 Poor data quality of case reports

 High number of cases with confounding factors / alternative explanations

 Signs of stimulated reporting e.g. increased media attention

 Abnormal reporting pattern

 Presence of other risk factors for the event: underlying disease, co-morbidities, co-med-
ications

Additional sources of information

Important sources: literature, other ADR databases, clinical trial data, registries, drug utilisation
data, studies

1. Additional data

 National databases (e.g. for signals detected in literature)

 Databases with larger datasets such as EudraVigilance or VigiBase (e.g. for signals de-
tected in national databases, literature or other sources)

 Information on active substances from the same class (e.g. for signals detected in national
database or Eudravigilance)

 Literature findings, regarding similar case reports, pharmacoepidemiological studies or

studies suggestive of a potential mechanism of action

 Clinical trial data

 Pre-clinical data

57
SCOPE Work Package 5
Signal Management – Best Practice Guide

2. Information on extent of exposure and characteristics of exposed patients

 Drug utilisation data (national exposure, patterns of use) from national drug utilisation reg-
istries or from PSURs

 Healthcare databases

3. Additional information

 RMP of the medicinal product for ongoing or planned studies that might provide evidence
on the association

 ENCePP registry for studies for ongoing or finalised studies, which might provide evi-
dence on the association

 Other registries

58
SCOPE Work Package 5
Signal Management – Best Practice Guide

Annex 3. Minimum information required for tracking of

signals within the MSs
Variables to be tracked

Administrative details
 Drug identification (active substance, Anatomic Therapeutic Chemical Classification (ATC),
trademark, drug class)

 ADR description (preferably MedDRA terminology)

 Signal ID: e.g. year, serial number or EPITT number,

 Regional centre involved (if applicable)

 Reference to other associated signals

Signal detection
 Source of information based on which the signal was initially detected

 Signal identification date (to be complemented with updates when status is reviewed)

 Number of case reports (if applicable)

Signal validation
 Date of signal validation

 Information taken into account for its validation

 Result (output) of the validation process

Decision making
 Date of expert/committee advice/opinion

 Expert/committee advice/opinion

59
SCOPE Work Package 5
Signal Management – Best Practice Guide

Annex 4. Signal detection methods and thresholds used in

various NCAs
Quantitative Method Number of Applied Thresholds
countries

Proportional Reporting 2 PRR lower bound 95% c.i. ≥ 1 & n ≥ 3 cases

Ratio (PRR)
1 PRR lower bound 95% c.i. ≥ 1

2 PRR lower bound 95% c.i. ≥ 2 & n ≥ 3 cases

2 PRR lower bound 95% c.i. ≥ 1 & n ≥ 5 cases3 *

1 PRR ≥ 3, PRR lower bound 95% c.i. ≥ 1 & n>2

Reporting Odds Ratio 1 ROR lower bound 95% c.i. ≥ 1

(ROR)

Information 1 IC lower bound 95% credibility interval > 0

Component (IC)

Empirical Bayes 1 EB05 ≥ 1.8 & n ≥ 3 & EBGM ≥ 2.5

Geometric Mean
(EBGM) 1 EB05 ≥ 1.8 and n>1 case **
* Three cases may be used for some drug-PT combinations
** Quantitative method applied only to serious reports (n=1)

3 In line with the results from the following publication: Slattery J, Alvarez Y, Hidalgo A. Choosing thresholds for
statistical signal detection with the proportional reporting ratio. Drug Saf. 2013 Aug;36(8):687-92.

60
SCOPE Work Package 5
Signal Management – Best Practice Guide

Annex 5. Regulatory Pharmacovigilance Prioritisation

System
Strength of evidence
 Disproportionality measure/risk estimate

 Data sources

 Evidence from RCT or meta-analysis

 Biological plausibility

Public health implications

 Drug/vaccine exposure

 Frequency of ADR

 Health consequences

 Spontaneous case reports

Agency regulatory obligations

 Recent parliamentary questions

 European obligations

 Ministerial/public health authority concern

 Marketing authorisation holder (MAH) application

Public perceptions
 Media attention

 Factors likely to cause public anxiety

 Public misperceptions

 Other public concern

61
SCOPE Work Package 5
Signal Management – Best Practice Guide

Annex 6. Glossary
Term Definition

Confirmed A signal that has been entered into EPITT and for which the confirmation
signal step performed either by the signal identifier, the LMS for that active
substance or the PRAC Rapporteur, has concluded that the signal should
be further discussed at PRAC for further prioritisation, analysis and
assessment, according to the IR.

European The NCAs from the 31 EEA Member States, together with the EMA and
Regulatory the European Commission (EC) constitute the European Medicines
Network Regulatory Network.

Lead Member For nationally authorised products approved in more than one MS, a
State worksharing has been organised whereby LMSs have been appointed to
monitor EudraVigilance data on behalf of the other MSs. The LMS is also
responsible for the validation and confirmation of signals in EPITT on
behalf of the other MSs for the respective nationally authorised products.
LMSs may also be supported in the fulfilment of its tasks by a co-leader.
When appointing a LMS, and as appropriate a co-leader, the CMD(h) in
collaboration with the PRAC, may take into account whether any MS is
acting as reference MS or as a Rapporteur for the assessment of PSURs
for the respective product.

Non-confirmed A signal that has been entered into EPITT, but for which, depending on
signal the origin of the signal, either the signal identifier, the LMS for that active
substance or the PRAC Rapporteur, has concluded that no discussion at
PRAC is warranted and therefore is not set into the PRAC agenda.

Non-validated A signal where the signal validation process of evaluating the data
signal supporting the detected signal has concluded that the available
documentation at that point in time does not contain sufficient evidence
demonstrating the existence of a new potentially causal association, or a
new aspect of a known association, and therefore a further analysis of the
signal is not warranted.

Reflected event An event for which there is sufficient information already included in the
(in the SmPC of currently approved SmPC of the active substance and therefore it is not
an active considered to be provide new information.
substance)

62
SCOPE Work Package 5
Signal Management – Best Practice Guide

Term Definition

Signal Information arising from one or multiple sources, including observations

and experiments, which suggests a new potentially causal association, or
a new aspect of a known association, between an intervention and an
event or set of related events, either adverse or beneficial, that is judged
to be of sufficient likelihood to justify verificatory action (IR Art 19(1)). A
signal generally involves an active substance regardless of its indication,
strength or route of administration and applies to all brand
names/medicinal products containing the active substance, including
fixed combinations. However, in some instances a signal may be relevant
only to a particular indication, strength or route of administration. On the
other hand, a signal may encompass all active substances of a
therapeutic class.

Signal The process of assessing all scientific evidence available. This might
assessment include pharmacological, non-clinical and clinical data and information
from other sources. This review should be as complete as possible
regarding the sources of information, including the application dossier,
literature articles, spontaneous reports, expert consultation, and
information held by MAHs and competent authorities.

Signal Procedural step through which a signal that has been entered into EPITT
confirmation is set for discussion in the PRAC agenda for further prioritisation, analysis
and assessment, according to the IR. Depending on the origin of the
signal, the confirmation step is performed within 30 days of its receipt by
the PRAC Rapporteur (for centrally authorised products), the LMS (if
appointed, for nationally authorised products) or by the signal identifier
(for nationally authorised products that have not yet been allocated to a
LMS).

Signal detection Process of monitoring post-marketing safety data for information that
would suggest a new potentially causal association, or a new aspect of a
known association between an intervention and an event or set of related
events, either adverse or beneficial, that is judged to be of sufficient
likelihood to justify verificatory action. It usually involves a combination of
statistical methods and review of individual case safety reports, as well as
any relevant source of information (e.g. scientific literature).

Signal identifier In the context of this BPG, the signal identifier is the regulatory authority
(EMA or MS) that detected or first became aware of the signal.

Signal Set of activities performed to determine whether, based on an

management examination of ICSRs, aggregated data from active surveillance systems
process or studies, literature information or other data sources, there are new risks
causally associated with an active substance or a medicinal product or
whether known risks have changed. It includes the following activities:
signal detection, signal validation, signal confirmation, signal analysis and
prioritisation, signal assessment and recommendation for action (IR Art
21(1)).

63
SCOPE Work Package 5
Signal Management – Best Practice Guide

Term Definition

Signal A continuous process performed through the whole signal management

prioritisation process, which aims to identify those signals with a potential important
public health impact or which may significantly affect the benefit-risk
profile of the medicinal product and thus require urgent attention and
management without delay. The prioritisation dictates the timeframe for
submission and assessment of data. While signal prioritisation is
described in the IR as a process whose responsibility falls on the PRAC
after a signal has been confirmed in EPITT, in this BPG signal prioritisation
is further explained as a continuous process that should be performed
during the whole signal management process, rather than a signal step,
and performed also at a national level, by the MSs.

Signal Process of evaluating the data supporting a detected signal in order to

validation verify that the available documentation contains sufficient evidence
demonstrating the existence of a new potentially causal association, or a
new aspect of a known association, and therefore justifies further analysis
of the signal (IR Art 21(1)).

Validated signal A signal where the signal validation process of evaluating the data
supporting the detected signal has verified that the available
documentation contains sufficient evidence demonstrating the existence
of a new potentially causal association, or a new aspect of a known
association, and therefore justifies further analysis of the signal (IR Art
21(1)).

64
SCOPE Work Package 5
Signal Management – Best Practice Guide

8. References

1. EMA. Guideline on good pharmacovigilance practices (GVP) Annex I -Definitions. 2012.

2. Regulation (EU) No 1235/2010 of the European Parliament and of the Council of 15

December 2010 amending, as regards pharmacovigilance of medicinal products for human
use, Regulation (EC) No 726/2004 laying down Community procedures for the
authorisation and supervision of medicinal products for human and veterinary use and
establishing a European Medicines Agency, and Regulation (EC) No 1394/2007 on
advanced therapy medicinal products.

3. Directive 2010/84/EU of the European Parliament and of the Council of 15 December 2010
amending, as regards pharmacovigilance, Directive 2001/83/EC on the community code
relating to medicinal products for human use.

4. Commission Implementing Regulation (EU) No 520/2012 (4) on the Performance of

Pharmacovigilance Activities Provided for in Regulation (EC) No 726/2004 and Directive
2001/83/EC. 2012;

5. EMA. Guideline on good pharmacovigilance practices (GVP), Module IX – Signal

management. 2012.

6. EMA. Questions & answers on signal management (Internet). (cited 2016 Jan 13). Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Other/2013/09/WC500150743.pdf

7. SCOPE WP5 Literature review (Internet). 2014. Available from:

http://www.scopejointaction.eu/_assets/files/2016-04-SCOPE-WP5-literature-review-
FINAL(1).pdf

8. CIOMS Working Group VIII. Practical aspects of signal detection in pharmacovigilance.

Geneva; 2010.

9. PROTECT website. (Internet). (cited 2013 Mar 22). Available from: http://www.imi-
protect.eu/

10. Wisniewski AFZ, Bate A, Bousquet C, Brueckner A, Candore G, Juhlin K, et al. Good Signal
Detection Practices: Evidence from IMI PROTECT. Drug Saf. 2016.

11. Meyboom RH, Egberts AC, Edwards IR, Hekster YA, de Koning FH, Gribnau FW. Principles
of signal detection in pharmacovigilance. Drug Saf. Int. J. Med. Toxicol. Drug Exp. 1997;
16:355–65.

12. 2015 Annual Report on EudraVigilance for the European Parliament, the Council and the
Commission Reporting period: 1 January to 31 December 2015 (Internet). Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Report/2016/03/WC500203705.pdf

13. Hauben M, Aronson JK. Defining “signal” and its subtypes in pharmacovigilance based on
a systematic review of previous definitions. Drug Saf. Int. J. Med. Toxicol. Drug Exp.
2009;32:99–110.

65
SCOPE Work Package 5
Signal Management – Best Practice Guide

14. The EU regulatory network incident management plan for medicines for human use-
EMA/351583/2015 (Internet). 2015. Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Other/2012/07/WC500130379.pdf

15. Seabroke S, Candore G, Juhlin K, Quarcoo N, Wisniewski A, Arani R, et al. Performance of

Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases. Drug
Saf. 2016.

16. Vaccines for prophylaxis against infectious diseases (Internet). 2016. Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/12/W
C500157839.pdf

17. Guideline on Biological Medicinal Products (Internet). Available from:

http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_gui
deline/2015/12/WC500198757.pdf

18. List of safety concerns from RMPs (Internet). http://www.hma.eu/464.html.

19. ENCePP Home Page (Internet). (cited 2016 Mar 28). Available from: http://www.encepp.eu/

20. Drug utilisation databases inventory (Internet). Available from:

http://www.imiprotect.eu/documents/DUinventory_2011_6_WORD97-2003.pdf.

21. List of active substances subject to worksharing for signal management (Internet). 2015.
Available from:
http://www.ema.europa.eu/ema/pages/includes/document/open_document.jsp%3FwebC
ontentId%3DWC500133308&rct=j&frm=1&q=&esrc=s&sa=U&ved=0ahUKEwiWlKGQzY7M
AhUGXQ8KHXfZCzUQFggbMAE&usg=AFQjCNEhPlpUHiH0B-1RlRAi6umehzNzdg

22. Swain E, Morgan S, Brewster W, Kauser S. Early communication of drug safety concerns:
a feasibility study on enhancing interaction between the pharmaceutical industry and
regulators. Pharmacoepidemiol. Drug Saf. 2010;19:232–7.

23. Pontes H, Clément M, Rollason V. Safety Signal Detection: The Relevance of Literature
Review. Drug Saf. Int. J. Med. Toxicol. Drug Exp. 2014.

24. Van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A
comparison of measures of disproportionality for signal detection in spontaneous reporting
systems for adverse drug reactions. Pharmacoepidemiol. Drug Saf. 2002;11:3–10.

25. Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of

Pharmacovigilance Signal-Detection Algorithms for the FDA Adverse Event Reporting
System. Clin Pharmacol Ther. 2013; 93:539–46.

26. Candore G, Juhlin K, Manlik K, Thakrar B, Quarcoo N, Seabroke S, et al. Comparison of

Statistical Signal Detection Methods Within and Across Spontaneous Reporting
Databases. Drug Saf. 2015;1–11.

27. Slattery SJ, Alvarez Y, Hidalgo A. Choosing Thresholds for Statistical Signal Detection with
the Proportional Reporting Ratio. Drug Saf. Int. J. Med. Toxicol. Drug Exp. 2013.

66
SCOPE Work Package 5
Signal Management – Best Practice Guide

28. Pearson RK, Hauben M, Goldsmith DI, Gould AL, Madigan D, O’Hara DJ, et al. Influence of
the MedDRA hierarchy on pharmacovigilance data mining results. Int. J. Med. Inf.
2009;78:e97–103.

29. Richard Hill, Johan Hopstadius, Magnus Lerch, G. Niklas Norén. An attempt to expedite
signal detection by grouping related adverse reaction terms. Proceedings of the First
Workshop on Computational Methods in Pharmacovigilance, Medical Informatics in
Europe (MIE) Conference. Pisa, Italy, 2012.

30. EMA. Guideline on good pharmacovigilance practices (GVP), Module VI Management and
reporting of adverse reactions to medicinal products. 2014.

31. Seabroke S, Wise L, Waller P. Development of a novel regulatory pharmacovigilance

prioritisation system: an evaluation of its performance at the UK Medicines and Healthcare
products Regulatory Agency. Drug Saf. 2013;36:1025–32.

32. FDA. Classifying Significant Postmarketing Drug Safety Issues (Internet). 2012. Available
from:
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidan
ces/UCM295211.pdf

33. Levitan B, Yee CL, Russo L, Bayney R, Thomas AP, Klincewicz SL. A model for decision
support in signal triage. Drug Saf. Int. J. Med. Toxicol. Drug Exp. 2008;31:727–35.

34. Segec A, Keller-Stanislawski B, Vermeer NS, Macchiarulo C, Straus SM, Hidalgo-Simon A,

et al. Strategy in Regulatory Decision-Making for Management of Progressive Multifocal
Leukoencephalopathy. Clin. Pharmacol. Ther. 2015;98:502–5.

35. Star K, Edwards IR. Pharmacovigilance for children’s sake. Drug Saf. 2014;37:91–8.

36. Midlöv P. MP. Pharmacokinetics and pharmacodynamics in the elderly. OA Elder. Med.
2013.

37. Maignen F, Hauben M, Hung E, Van Holle L, Dogne J-M. Assessing the extent and impact
of the masking effect of disproportionality analyses on two spontaneous reporting systems
databases: EXTENT AND IMPACT OF MASKING IN SRS DATABASES.
Pharmacoepidemiol. Drug Saf. 2014;23:195–207.

38. MedDRA paediatric and gender alert term lists (Internet). Available from:
http://www.meddra.org/paediatric-and-gender-adverse-event-term-lists.

39. Giezen TJ, Mantel-Teeuwisse AK, Straus SMJM, Schellekens H, Leufkens HGM, Egberts
ACG. Safety-related regulatory actions for biologicals approved in the United States and
the European Union. JAMA J. Am. Med. Assoc. 2008;300:1887–96.

40. Trifirò G, Pariente A, Coloma PM, Kors JA, Polimeni G, Miremont-Salamé G, et al. Data
mining on electronic health record databases for signal detection in pharmacovigilance:
which events to monitor? Pharmacoepidemiol. Drug Saf. 2009;18:1176–84.

67
SCOPE Work Package 5
Signal Management – Best Practice Guide

41. Important Medical Events List (Internet). Available from:

http://www.ema.europa.eu/ema/pages/includes/document/open_document.jsp?webConte
ntId=WC500208836

42. PRAC Minutes of the meeting 7-10 October 2013 (Internet). Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Minutes/2013/11/WC50015442
4.pdf

43. EMA. Good Practice Guide Medication Error (Internet). Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_gui
deline/2015/11/WC500196979.pdf

44. PRAC Minutes of the meeting of 6-9 January 2014 (Internet). Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Minutes/2014/02/WC50016189
2.pdf

45. Caster O, Juhlin K, Watson S, Norén GN. Improved Statistical Signal Detection in
Pharmacovigilance by Combining Multiple Strength-of-Evidence Aspects in vigiRank :
Retrospective Evaluation against Emerging Safety Signals. Drug Saf. Int. J. Med. Toxicol.
Drug Exp. 2014;37:617–28.

46. Nair Gj. Ensuring quality in the coding process: A key differentiator for the accurate
interpretation of safety data. Perspect. Clin. Res. 2013;4:181.

47. MedDRA Term Selection: points to consider, based on MedDRA Version 19.0, March 2016
(Internet). Available from: http://www.meddra.org/sites/default/files/guidance/file/9491-
1900_termselptc_r4_11_mar2016.pdf

48. MedDRA Data retrieval and presentations: points to consider, based on MedDRA Version
19.0, March 2016 (Internet). Available from:
http://www.meddra.org/sites/default/files/guidance/file/9610-
1900_datretptc_r3_11_mar2016.pdf

49. EMA. Guideline on good pharmacovigilance practice (GVP), Module I Pharmacovigilance

systems and their quality systems. 2012.

50. PROTECT Adverse Drug Reactions Database (Internet). Available from: http://www.imi-
protect.eu/adverseDrugReactions.shtml

68