Systematic review and meta

analysis
 systematic review and meta analysis
• The ability to critically evaluate
literature and weigh evidence is
fundamental to evidence-based
healthcare. This task is
becoming increasingly difficult,
as the amount of information is
growing exponentially from year
to year. So, for example,
compared with 1991, in 2008,
three, eight and forty times
more articles were indexed on
the problems of malaria, obesity
and biodiversity, respectively[1].
[1] Pautasso M (2010) Worsening file-drawer problem in the
abstracts of natural, medical and social science databases.
Scientometrics 85: 193–202. do
Hierarchy of evidence
 What is a systematic review?

• “A systematic review is a review in which there is a comprehensive

search for relevant studies on a specific topic, and those identified are
then appraised and synthesized according to a predetermined and
explicit method.” (Klassen 1998)

• A systematic review attempts to collate all empirical evidence that fits

pre-specified eligibility criteria in order to answer a specific research
question. It uses explicit, systematic methods that are selected with a
view to minimizing bias, thus providing more reliable findings from
which conclusions can be drawn and decisions made (Antman 1992,
Oxman 1993)
https://ru.wikipedia.org/wiki/Систематический_обзор
Systematic reviews vs ‘Traditional’ reviews

‘Traditional’ reviews Systematic reviews

Unclear how included studies were Protocol specifies exact research
selected question/inclusion criteria
Based on incomplete set of studies Comprehensive search to find all
relevant studies
May not consider quality of included Standardised critical appraisal across
studies included studies
Unclear how conclusions follow from Methods transparent and reproducible
included studies
 Why use systematic
reviews?
• Minimise the impact of bias/errors
• Can help to end confusion
• Highlight where there is not sufficient evidence
• Combining findings from different studies can highlight new
findings
• Can mitigate the need for further trials
Who undertakes systematic
reviews?
• Multidisciplinary teams
– Clinicians
– Health services researchers
– Information scientists
– Statisticians
– Health economists
– Patient and public involvement
Systematic Reviews - the
process (1)
• Define review question – very precisely; in partnership
with commissioners, clinicians, patients (as appropriate)
• Develop protocol – provides transparency; defines exact
inclusion criteria and methods
• Identify relevant studies – usually comprehensive search
across multiple bibliographic databases plus reference
checking
• Assess eligibility – careful matching of studies against
inclusion criteria
Systematic Reviews - the
process (2)
• Extract relevant data – only what is required to answer
the question
• Critically appraise studies – use a published tool to
compare methodological features across studies
• Synthesise appropriately – depending on type of question
and study designs quantitative or qualitative synthesis
• Disseminate to appropriate audience – full transparent
write-up plus as appropriate short report to funder, journal
article, patient leaflet…
Formulating review question
• Questions may be broad or narrow
• Well-formulated questions will guide many aspects of
the review process
– Searching strategy
– Inclusion/exclusion criteria
– Data extraction
– Choice of synthesis method
– Presentation/dissemination of findings
 Formulating review question
Advantages Disadvantages
Broad Comprehensive summary of May ‘lump’ interventions that are
question all evidence heterogeneous – apples/ pears

Findings are likely to be May be a lot of studies to manage

generalisable (difficult for narrative synthesis)

Narrow More discrete and May be too few studies to provide

question manageable inclusion meaningful result (although this can
criteria be useful evidence of ‘gap’)

Less heterogeneity May not be generalizable at all

 Formulating review
question: tools (1)
PICO is for effectiveness questions which usually use controlled trials
 Formulating review question: tools (2)

SPICE is designed for questions around public health interventions

1. Setting what is the context of the question?

e.g. European countries

2. Perspective who are the users/potential users of the outcomes?

e.g. chronically ill

3. Intervention what is being done to them?

e.g. increased cost-sharing

4. Comparison what are the alternatives?

e.g. no increase

5. Evaluation how will you measure if the intervention is successful?

e.g. access to health services
 Formulating review
question: tools (3)
Sample a description of what is likely to be a smaller group of participants

Phenomenon (of) may not be an intervention or exposure

Interest

Design details of the design will help to make decisions about the
robustness of the research

Evaluation what are you measuring?

Research type qualitative, mixed methods?

Any questions??
Developing protocol
• A protocol is an essential component of the systematic
review process
• Helps to ensure careful a priori planning of whole process
– Consistency
– Transparency
– Integrity
• Journals often require a protocol registration number before
publishing systematic reviews
 Developing protocol
• One of the features that distinguish a systematic
review from a traditional review is the pre-specification
of criteria and methods
– What studies will be included and excluded? (use previous
question formulation tools)
– How (exactly) will each stage be conducted? (search strategy –
consult librarian, study selection, critical appraisal, synthesis
method – consult statistician)
– What are the primary and secondary outcomes of interest?
PROSPERO – CRD initiative

• Search for ongoing reviews

• Register planned review online (& update as work progresses)
• Avoids duplication of reviews
Searching for eligible studies
• Can use the same tools to help define search as for
question formulation (PICO, SPICE, SPIDER)
– Use only the items needed to define search (usually at minimum population
and intervention/phenomenon of interest), sometimes outcomes
– Consult information specialist or librarian for advice (the search is the first
potential major source of bias in your review)
Searching for eligible studies
• Thesaurus headings (e.g. MeSH in MEDLINE)
• Controlled vocabulary thesaurus used for indexing articles
• Standardised search terms regardless of how author has described the study
• Each database has a different thesaurus – must translate
Searching for eligible studies
• Where to search
• Electronic databases according to the topic area: MEDLINE, EMBASE, Cochrane databases,
PsycInfo, etc.
• Grey literature, dissertations, theses, conference proceedings, national bodies (NICE, HTA), clinical
trial database (www.clincialtrails.gov/)
• University library web pages give advice about databases available in different topic areas
• Check existing systematic reviews in a similar area for ideas
• Look at the databases own guidance for searching they vary!
• Download results to reference management software (e.g. University supports Endnote,
alternatively Mendeley)
Boolean operators
 Selecting included studies
• Select potentially eligible abstracts (in Endnote)
• Assess titles/abstracts against your predetermined inclusion/exclusion criteria, use
groups or custom fields to record
• Retrieve full text articles of initial selections
• Assess full text for inclusion
• Requires judgement (>1 reviewer)
• Check reviewer agreement (3rd review to resolve)
• Use a selection form to ensure consistency and record decisions
Data Extraction (1)
• Pilot a data extraction form
– See options later
• Collect the following information:
– Citation details to identify the study
– Characteristics of population/intervention
– Outcomes at baseline and follow-up
– Methodological details according to chosen critical appraisal tool
Data Extraction (2)
• What effect measures do you want to calculate?
– What data do you need to do this?
• How are you planning to group studies for the analysis?
– By intervention, study design??
• What information do you need to extract to enable you to
organise and analyse the way you want?
Data Extraction (3)
• Extract sufficient information:
– to describe studies
– to allow you to undertake the planned analysis
– so you do not need to return to the full text papers
• But not too much
– Don’t waste time extracting ‘nice to have’ but unnecessary detail

REMEMBER YOUR PROTOCOL

IT IS YOUR ROADMAP, FOLLOW IT!
 Consistency/Standardisation

• Data extractors need to collect and interpret data in a

standard way
• Essential >one reviewer is extracting data
• Independent piloting of data extraction forms – always one
standardised form
• Regular discussion of progress/disagreements
• Regular comparison of data extraction – don’t wait till the
end
 Critical Appraisal
• Why - what are we trying to achieve?
• Not all published and unpublished literature is rigorous!
– being in a journal doesn’t mean it is good
• Quality may be used as an explanation for differences
in study results or to guide interpretation of findings,
strength of inferences
Data Synthesis
• Building up; putting together; making a whole out of the parts; the
combination of separate elements of thought into a whole; reasoning
from principles to a conclusion.
Meta-analysis/forest plot
• Qualitative studies
• Three broad categories
• Rigour: has a thorough and appropriate approach been applied to key
research methods in the study?
• Credibility: are the findings well presented and meaningful?
• Relevance: how useful are the findings to you and your organisation?
 Meta analysis
• Meta-analysis - the concept of scientific methodology. It means
combining the results of several studies using statistical methods to
test one or more interrelated scientific hypotheses.

https://ru.wikipedia.org/wiki/Метаанализ
 Meta analysis
• Summary analysis of several original articles (secondary statistical
analysis)
• At the moment, it has become much easier to execute, since several
packages for the R system have been developed:rmeta
• meta
• netmeta
• metafor
• bspmma
 Meta analysis
•Furberg and Morgan (1987) identify six reasons for conducting a meta-
analysis:
•Getting a more stable assessment of the effect of therapy
•Assessment of differences between studies and the possibility of generalizing
the results
•Analysis of the effect of therapy in individual subgroups
•Providing more reliable data to licensing organizations to obtain permission
to use the drug
•Needs Assessment and Research Planning
•Comparison of the results of this study with others conducted on this topic.
 Meta analysis
•Coding scheme
•Publication
•Year
•A source
•Methods
•Sample Type
•Additional data
•Statistics, summary data
 Meta analysis
•Particular attention to the size of the effect:
•The drug A. t (256) = 4.0, p <0.001
•Drug B. t (64) = 2.0, p <0.05
•Drug B. t (4) = 0.5, p <0.64
 Effect size

• According to Cohen (1988), if the size of the

effect does not exceed 0.2, they speak of a
weak effect of therapy, if it turns out to be
0.5, they talk about the effect of medium
strength and if it exceeds 0.8, then they say
about the great effect of the drug.
•Thanks for your attention!