What Is the Difference Between

a Systematic Review
37
and a Meta-analysis?

Shakib Akhter, Thierry Pauyo, and Moin Khan

37.1 Hierarchy of Evidence ical impact [2, 3, 27]. Higher-quality research

evidence (such as level 1 and level 2) is readily
The hierarchy of evidence serves as the founda- brought from bench to bedside as their superior
tion for evidence-based practice and provides a methods increase study validity. But the question
top-down descriptive visualization of the best arises: where do systematic reviews and meta-­
available evidence. The level of evidence is pro- analysis fall in this hierarchy of evidence? The
portional to reliability, quality, and validity; the answer is that they each fall in every level. A sys-
higher these factors, the higher the study lies in tematic review and meta-analysis of well-done
the hierarchy of evidence [2, 27]. Clinicians and high-quality randomized controlled trials are
scientists seek higher levels of evidence as these considered the pinnacle of evidence-based
studies have the greatest potential impact for research. It is important to understand and appre-
clinical practice [3]. Experimental study designs, ciate that systematic reviews and meta-analyses
such as randomized controlled trials, occupy the are not always atop the hierarchy of evidence; if
highest level of evidence (level 1 evidence) in a researcher conducts a systematic review of
research, followed by observational study level 3 evidence, the review will be considered
designs, such as cohort studies (level 2 evidence) level 3 evidence. Conversely, a systematic review
[2, 3, 27]. These research designs are followed by of level 1 evidence will remain level 1 evidence.
case control studies (level 3 evidence), case stud- Accordingly, the study design and level of evi-
ies and case series (level 4 evidence), and finally dence are directly proportional. Systematic
expert opinions (level 5 evidence) [2, 3, 27]. reviews and meta-analyses remain confined to
Clinicians should cautiously apply low-quality the level of evidence they are used with but pro-
evidence such as the latter designs given their vide invaluable results and have tremendous clin-
poor reliability, reproducibility, validity, and clin- ical care implications.

S. Akhter (*) · M. Khan

Department of Orthopaedic Surgery, 37.2  hy Perform a Systematic
W
McMaster University, Hamilton, ON, Canada Review or Meta-analysis?
Department of Health, Evidence, and Impact,
McMaster University, Hamilton, ON, Canada With over 50 million scholarly articles published
e-mail: [email protected] to date, difficulty exists among clinicians and sci-
T. Pauyo entists in organizing and understanding the vast
Department of Orthopaedic Surgery, amounts of available literature [10]. Arguably,
McGill University, Montreal, QC, Canada

© ISAKOS 2019 331

V. Musahl et al. (eds.), Basic Methods Handbook for Clinical Orthopaedic Research,
https://doi.org/10.1007/978-3-662-58254-1_37
332 S. Akhter et al.

the most effective and efficient method in synthe- e­ ligibility criteria, are reported. Narrative reviews
sizing available data in order to make evidence-­ employ a qualitative approach to critically
based decisions is by conducting systematic appraise literature from a specific context or the-
reviews and meta-analyses. These methods iden- oretical perspective. A narrative review is highly
tify, critically appraise, and evaluate several stud- susceptible to bias and lacks reproducibility.
ies pertaining to a single prespecified research Researchers supplement narrative reviews with
question [21]. Both methodologies allow expert intuitive and experiential evidence, reflect-
researchers to combine large portions of litera- ing the qualitative approach [20]. The lack of a
ture and produce results that are widely general- systematic process predisposes this type of
izable. Additional benefits include the ability to review to bias, specifically subjective selection
limit biases found in individual studies, which bias, which has implications on the validity and
increases the reliability of the findings. Although generalizability of the study.
the systematic review and meta-analysis method- A systematic review is governed by carefully
ologies synergistically analyse numerous studies, constructed stages which guide the cumulative
they both serve different yet complementary search method, screening, reviewing, catalogu-
roles. Systematic reviews are the cornerstone in ing, and reporting process of the selected studies
evidence-based medicine as the meticulous, to answer a research question of interest. The
exhaustive, systematic, and structured approach search is intended to provide an exhaustive
is effective in summarizing and critically analys- review of the current literature, which captures
ing the findings of relevant literature pertinent to all appropriate studies ensuring the research
a research questions. A meta-analysis may be question to be answered to its fullest extent. The
conducted in conjunction with a systematic carefully crafted and exhaustive methodology
review to enhance the credence of findings by minimizes bias and provides reliable and accu-
increasing the level of evidence. Through com- rate conclusions to be drawn. It has facilitated an
bining and analysing several studies, researchers improved dissemination of information to health-
aim to extrapolate results that more accurately care providers, the public, policymakers, and the
reflect true effects. Synthesizing data from mul- scientific community. A notable benefit is the
tiple studies allows researchers to achieve a resulting mitigation in the delay in bringing
greater level of statistical power, which is pre- research from bench to bedside leading to well-­
cluded in individual studies. Researchers practis- informed policy decisions and direction for future
ing this methodology are encouraged to follow research. Although systematic reviews rank high
criteria outlined by research quality improvement in the hierarchical chain of evidence, they are not
bodies, such as the Preferred Reporting Items for without flaws. Depending on the construction of
Systematic Reviews and Meta-analysis the search and screening criteria, a researcher
(PRISMA) statement [16]. may inadvertently exclude relevant studies.
However, given the systematic, explicit, and
comprehensive methodology of the review, the
37.3 Systematic Review risk of missing relevant studies is decreased in
comparison to a narrative review. Additionally,
37.3.1 What Is a Systematic Review? the generalizability of the findings may be lim-
ited as patients included in the review must be
Literature reviews are commonly classified as homogenous to the target population the
either a narrative or systematic review. A narra- researcher attempts to generalize. Heterogeneity
tive review is a synthesis of literature in a specific is a marked concern among researchers as a bal-
field constructed using a specific contextual or ance between internal and external validity must
theoretical point of view. The method is not be prioritized; although stringent eligibility crite-
nearly as robust or rigorous as no methodological ria produce homogenous data, generalizability is
approaches, such as the search strategy or affected as patients with differing characteristics
37 What Is the Difference Between a Systematic Review and a Meta-analysis? 333

may be excluded [28]. The systematic review may have significant implications to the validity
methodology is indicated in this context as it can of the review. An extensive search without lan-
provide a descriptive and analytical summary of guage restrictions is recommended [12]. A com-
heterogeneous and dissimilar studies, which usu- prehensive search often has three or more online
ally preclude the use of a meta-analysis [28]. databases included. Examples of commonly used
databases include MEDLINE, EMBASE,
CENTRAL, CINAHL, PUBMED, and others.
37.3.2 A
 Guide to Performing These platforms allow the researcher to conduct a
a Systematic Review comprehensive and exhaustive search of litera-
ture published on the World Wide Web, download
Six essential steps in conducting a systematic results, and perform a citation analysis [17]. A
review are described. critical tool for success, although not necessary,
Step 1: The Research Question—The first and but highly recommended, is seeking a profes-
foremost step in the process of performing qual- sional librarian to assist in the search strategy
ity research is formulating appropriate research development and implementation [25]. The terms
questions. Commonly underestimated, formulat- should also be reflective of the PICO format to
ing the research question requires careful consid- produce an effective search strategy that identi-
eration of a multitude of factors and can fies all relevant studies and balances sensitivity
potentially consume a notable amount of time. and specificity [25]. In this context, good sensi-
The research question should include the prob- tivity and specificity are reached when the search
lem or question of interest, population of interest, produces a high number of relevant studies and a
intervention, and comparator, along with the out- low number of irrelevant studies, respectively.
comes of interest [1]. Using the population, inter- With an increasing trend in technological domi-
vention, comparator, and outcomes (PICO) nance in research, many authors chose to solely
format helps ensure the question is direct and rel- search electronic databases, but searches of refer-
evant. Although sometimes loosely stated, ence lists on articles and specific journals can
authors should explicitly present the question in a also be done [25]. A search of not only published
structured format as it allows consumers to data via online databases but also of ‘grey litera-
directly understand the goal of the project [12]. ture’ decreases the risk for potential publication
Step 2: Constructing Eligibility Criteria— bias of results. Grey literature refers to scholarly
Selecting appropriate eligibility criteria is funda- literature that is not formally published and can
mental in ensuring the most relevant and include dissertations, policy documents, book
appropriate studies are included in the review. chapters, and conference abstracts, research
Researchers can include all key components reports, and unpublished research data [9].
needed to comprehensively answer their question Step 4: Screening, Selection, and Extraction—
if the criteria are reflective of the PICO points This process begins upon completion of the
identified in Step 1. Additionally, researchers search. A list of all article abstracts that appeared
should clearly identify which type of studies they in the search undergoes an abstract screening, in
are interested in (i.e. randomized trials, cohort which researchers make decisions about includ-
studies, etc.) and other operational factors rele- ing or excluding studies based on their relevance
vant to the studies including, but not limited to, to the eligibility criteria elicited from the
year published, language, and number of partici- abstract. A recommended practice to establish
pants [25]. inter-rater reliability and increase validity of the
Step 3: Constructing the Search Strategy— study is to have two independent reviewers con-
Using appropriate search terms is essential in duct this process independently. Often, review-
ensuring an exhaustive search of literature rele- ers may face disagreements whether to include
vant to the research question. An incomprehen- or exclude studies. Attempts to reach agreements
sive search can exclude important studies that between the two reviewers should be made, but
334 S. Akhter et al.

if ­unsuccessful, another study researcher should [28]. The data extraction p­ rocess also serves as
be consulted. Researchers should log all activi- the final screening process, as the extracted data
ties in this process, such as which studies were will guide final decisions regarding which stud-
included and excluded and the reasons for each ies will be included or excluded [25, 28].
[25]. This log will help researchers create a flow Researchers must also be prepared for when they
diagram to visually represent articles included. face studies that have missing or incomplete
Figure 37.1 is an example taken from the full- data, for which they must contact the individual
text article of Clinical Vignette 2. Once all titles study authors to obtain [16].
and abstracts are screened and a significantly Step 5: Quality Assessment—Appraising the
shorter list of studies remains, a second full-text study quality, although not required, is highly
screening process should be conducted by the recommended to produce a comprehensive and
same reviewers to ensure reliability and consis- highly valid systematic review. A standardized
tency in study methods. This process will operational definition of ‘study quality’ is elu-
exclude all remaining irrelevant studies and sive but generally refers to the confidence a
identify which have extremely high potential to researcher holds in the study’s design and meth-
be included in the review. Finally, researchers ods to minimizing bias [19, 21, 28]. In other
should create a data table or a standardized data words, a quality assessment is a critical appraisal
extraction form to systematically organize and to determine if the design, conduct, and methods
extract data. This table or form is unique to each of a study will reduce systematic error and bias,
systematic review and may include patient clini- which in turn has implications on its internal and
cal and demographic characteristics, author external validity [4, 28]. Bias is defined by the
names, type of study design, and outcomes. It is Cochrane Collaboration as a deviation from truth
most commonly electronic (i.e. a constructed or a systematic error that can lead to either an
table in excel) given its associated considerable under- or overestimation of the true effect [6].
efficiency and mitigation of data errors due to There are multiple types of bias (selection,
mismanagement, but paper also may be used detection, reporting, attrition, performance
biases) that range from small to substantial and
can markedly limit the generalizability of a
Articles identified
by electronic research study [6]. However, researchers should
search n = 944 be cautious in the interpretation of a quality
Duplicates excluded n = 410 assessment as the process is inherently limited
by factors such as a lack of information provided
Titles and
abstracts screened
by the author as well as the absence of a ‘gold
n = 534 standard’ to reference level of quality [21, 26].
Articles excluded n = 528 Nonetheless, recommended guidelines such as
• No meniscal pathology n = 146
• No arthroscopic intervention n = 88
the five-point Oxford Quality Rating Scale or the
• No nonoperative comparison n = 139 Consolidated Standards of Reporting Trials
• Not degenerative tears n = 13 (CONSORT statement) are available for com-
• Not RCTs n = 142
Full texts prehensive quality assessments and should be
screened n = 6
used when appropriate. It is also recommended
Manual search of references that at least two independent reviewers conduct
• Articles included n = 2 the quality assessment process and increase
inter-rater reliability and study validity [8]. A
Article excluded
• Lack of outcomes reported n = 1
consensus within available literature outlines
four main biases that affect study quality: perfor-
Studies
included n = 7
mance, selection, attrition, and detection bias [4,
13, 24, 28]. The quality assessment is a critical
Fig. 37.1 Article selection flow diagram step in the systematic review process as these
37 What Is the Difference Between a Systematic Review and a Meta-analysis? 335

biases can lead to under- or ­overestimations of strong external validity through inter-reviewer
the true effect, and the resulting spurious asso- agreement, reliability via the kappa coefficient,
ciations may have detrimental implications for and consistency via Cronbach’s alpha coefficient
clinical practice [12, 28]. [23]. The kappa coefficient is a commonly used
Quality assessment tools depend on what type statistical measure of inter-rater reliability, and
of evidence is being synthesized. The Cochrane Cronbach’s alpha coefficient elicits the consis-
risk of bias (ROB) tool is considered the gold tency and correlation of the items of an instru-
standard for the quality assessment of random- ment, such as a survey, to determine its overall
ized controlled trials, whereas the methodologi- reliability [18, 22].
cal index for non-randomized studies (MINORS) Providing an overall confidence in the esti-
is one of many available measures for assessing mate of effect of a particular treatment or inter-
observational studies. The Cochrane ROB tool vention is increasingly being performed utilizing
ensures a systematic and critical appraisal of all the Grading of Recommendations, Assessment,
possible bias domains including selection, per- Development and Evaluations (GRADE) criteria.
formance, detection, attrition, reporting, and This GRADE method provides a reliable esti-
other forms of biases [29]. Assessing for these mate of the effect of the evidence across all stud-
potential biases ensures the researcher takes into ies for outcomes, as opposed to individual study
account systematic differences in baseline char- evaluation. An example of one of the factors cri-
acteristics, provision of care, outcome ascertain- tiqued by GRADE is shown in Fig. 37.3.
ment, participant withdrawals, and reported and Methodological flaws, treatment effects, consis-
unreported findings within the studies. The tency, and generalizability comprise the main
MINORS is a 12-item tool that is used for, but assessment domains seen in this tool [5]. Great
not limited to, quality assessment of systematic care should be taken to select the appropriate tool
reviews that employ observational studies for quality assessment and should be reflective of
(Fig. 37.2) [23]. This tool has demonstrated the type of studies being reviewed.

Methodological items for non-randomized studies Score†

1. A clearly stated aim: the question addressed should be precise and relevant in the light of available literature
2. Inclusion of consecutive patients: all patients potentially fit for inclusion (satisfying the criteria for inclusion) have been
included in the study during the study period (no exclusion or details about the reasons for exclusion)
3. Prospective collection of data: data were collected according to a protocol established before the beginning of the study
4. Endpoints appropriate to the aim of the study: unambiguous explanation of the criteria used to evaluate the main outcome
which should be in accordance with the question addressed by the study. Also, the endpoints should be assessed on an
intention-to-treat basis.
5. Unbiased assessment of the study endpoint: blind evaluation of objective endpoints and double-blind evaluation of subjective
endpoints. Otherwise the reasons for not blinding should be stated
6. Follow-up period appropriate to the aim of the study: the follow-up should be sufficiently long to allow the assessment of
the main endpoint and possible adverse events
7. Loss to follow up less than 5%: all patients should be included in the follow up. Otherwise, the proportion lost to follow up
should not exceed the proportion experiencing the major endpoint
8. Prospective calculation of the study size: information of the size of detectable difference of interest with a calculation of
95% confidence interval, according to the expected incidence of the outcome event, and information about the level for
statistical significance and estimates of power when comparing the outcomes
Additional criteria in the case of comparative study
9. An adequate control group: having a gold standard diagnostic test or therapeutic intervention recognized as the optimal
intervention according to the available published data
10. Contemporary groups: control and studied group should be managed during the same time period (no historical comparison)
11. Baseline equivalence of groups: the groups should be similar regarding the criteria other than the studied endpoints. Absence
of confounding factors that could bias the interpretation of the results
12. Adequate statistical analyses: whether the statistics were in accordance with the type of study with calculation of confidence
intervals or relative risk

Each domain is scored from 0-2. Items will either be scored as 0 (not reported), 1 (reported but
inadequate), or 2 (reported and adequate) [29]. A score for non-comparative studies of 16 and
comparative studies of 24 is globally accepted [29].

Fig. 37.2 The validated and updated version of the MINORS questionnaire [29]
336 S. Akhter et al.

Risk of bias Across studies Interpretation Considerations GRADE

assessment of
study limitations

Low risk of Most information is Plausible bias No apparent limitations. No serious

bias. from studies at low unlikely to limitations, do not
risk of bias. seriously alter downgrade.
the results.

Unclear risk Most information is Plausible bias Potential limitations are No serious
of bias. from studies at low that raised some unlikely to lower limitations, do not
or unclear risk of doubt about the confidence in the estimate downgrade.
bias. results. of effect.

Potential limitations are Serious

likely to lower confidence limitations.
in the estimate of effect. downgrade one
level.

High risk of The proportion of Plausible bias Crucial limitation for one Serious
bias. information from that seriously criterion, or some limitations.
studies at high risk weakens limitations for multiple downgrade one
of bias is sufficient confidence in the criteria, sufficient to lower level.
to affect the results. confidence in the estimate
interpretation of of effect.
results.
Crucial limitation for one Very serious
or more criteria sufficient limitations,
to substantiallly lower downgrade two
confidence in the estimate levels.
of effect.

Adopted from the Cochrane Handbook of Systematic Reviews of Interventions, this is an extension
of factor 1 of 5 of GRADE that extends the risk of bias assessment to make conclusions regarding
the limitations relating to outcomes [6].

Fig. 37.3 Factor 1 of 5 in GRADE assessment [6]

A ­comprehensive quality assessment adds to the methods, biases, quality assessment, total num-
review’s generalizability and validity and there- ber of patients in treatment and control groups,
fore leads to a greater clinical impact. intervention(s), control, outcomes, and any other
Step 6: Data Analysis and Interpretation of relevant information. The inclusion of items in
Results—The final steps of data analysis and the table should reflect the research question
interpretation should be conducted following a [20]. Interestingly, these information-rich tables
comprehensive quality assessment [4, 13, 24, provide sufficient information for the researcher
28]. This step requires the researcher to create a to determine if statistically pooling the data for a
concise and simple descriptive summary of each meta-analysis is possible [20]. If the study
included study [28]. Many researchers choose to includes sufficient data to be meta-analysed, that
display study characteristics in a tabular format. approach will be adopted as it is a higher level of
An example can be found in Table 1 in the full evidence. As a result, a systematic review can be,
article of the Clinical Vignette 1. The descriptive and is commonly, accompanied by a meta-­
table is a comprehensive summary of study char- analysis (Clinical Vignette 2).
acteristics and therefore should include, but is not When interpreting results, the researcher
limited to, author name, year published, study should carefully extrapolate conclusions based
37 What Is the Difference Between a Systematic Review and a Meta-analysis? 337

on the summarization and analysis of the included their research question of interest in the back-
studies. In this section, the findings must be ground section (Step 1). The eligibility criteria
briefly reiterated, and the final impressions from and the use of two independent reviewers were
this synthesis of the best available evidence highlighted (Step 2). Four databases were
should be explicitly stated. Here, the researcher searched producing 283 articles, of which 29
should highlight the strengths and limitations of were included following the screening process
their study and indicate how these findings may (Step 4). Results and figures for study and
or may not have clinical implications. Suggestions patient characteristics, along with results for
for future directions of research should also be study quality and clinical outcomes, were
included as they are valuable statements in sys- reported (Step 5). This information was also
tematic reviews, provided this methodology sum- visually displayed using histograms. The results
marizes available evidence pertaining to a specific in each individual studies were reported cumu-
field and highlights lacking areas. Systematic latively as a single article of evidence, without
reviews directly improve patient care as they the use of statistical methods. The authors
provide information that is a requisite for
­ finally critically interpreted the results in the
evidence-­based decision-making. discussion section (Step 6). Evidently, the
Clinical Vignette 1 displays a systematic authors clearly report their methods allowing
review on the efficacy of antifibrinolytic therapy reproducibility and demonstrating a strong
in reducing patient transfusions in orthopaedic internal validity in their review.
surgery. The authors clearly and explicitly state

Clinical Vignette 1: Systematic Review [11]

338 S. Akhter et al.

meaningful results, for example, by detecting

Fact Check modest associations. This design also provides
–– A systematic review is a comprehensive researchers a comprehensive understanding of
and exhaustive review of relevant litera- the true effect. Clinical, methodological, and sta-
ture specific to a topic or research tistical heterogeneities are issues of variability
question. that impose limitations on the systematic review
–– Systematic reviews may employ quali- and meta-analysis methodologies. With limited
tative or quantitative methods in review- heterogeneity, more data provides a more precise
ing and reporting homogenous or estimate of effect. Moreover, the utilization of
heterogeneous studies. statistical methods addresses notable concerns of
–– Compared to the meta-analysis method, generalizability and bias, which are inherent in
a systematic review can be completed the systematic review alone. With respect to het-
readily and requires less formal training, erogeneity, the meta-analysis quantifies between
but is of a lower level of evidence. group differences (differences between studies)
–– The six steps in performing a compre- and also explains them (e.g. using tools such as a
hensive systematic review include (1) meta-regression). However, the presence of
formulating an appropriate research excessive heterogeneity may result in faulty and
question; (2) formulating appropriate misleading conclusions. Statistical methods to
eligibility criteria; (3) forming and con- test for heterogeneity are discussed below.
ducting an appropriate search strategy; Although this methodology occupies the top of
(4) screening, selection, and extraction the hierarchy of evidence, as every research
of relevant results; (5) quality assess- design, it has limitations [7]. The nature of the
ment of the included studies; and (6) meta-analysis requires a large number of studies
critical appraisal to summarize and in order for an effect to be seen. Additionally,
interpret the findings. although it can combine the data of various stud-
ies, the meta-analysis is not capable of adjusting
for poor methodology of the studies included
which may skew the outcomes. It is important for
37.4 Meta-analysis the researcher to create specific eligibility crite-
ria, so that the literature search and review pro-
37.4.1 What Is a Meta-analysis? cess is maximally exhaustive and produces
methodologically sound studies.
A meta-analysis (Clinical Vignette 2), much like
a systematic review and often an extension of
one, also hinges on a systematic and exhaustive 37.4.2 A
 Guide to Performing
search of the literature. A meta-analysis differs a Meta-analysis
from a systematic review in that instead of simply
collecting and analysing the data, it employs sta- The six steps in performing a meta-analysis are
tistical methods to quantitatively synthesize the the same in performing a systematic review. The
results from multiple studies [15, 28]. This design only difference is additional methods are utilized
aims to expose true effects buried within data by to perform a meta-analysis in the data analysis in
analysing patterns to compare and contrast find- Step 6. The steps follow this chronological order:
ings of several studies. A meta-analysis has many (1) formulating an appropriate research question;
inherent benefits compared with a systematic (2) formulating appropriate eligibility criteria;
review. A notable advantage is that pooling stud- (3) forming and conducting an appropriate search
ies increases statistical power, otherwise unat- strategy; (4) screening, selection, and extraction
tainable in individual studies, leading to more of relevant results; (5) quality assessment of the
37 What Is the Difference Between a Systematic Review and a Meta-analysis? 339

included studies; and (6) analysing results and r­ aters (data collectors) agree in their independent
interpreting the findings. measurements [18]. A multitude of statistical
Step 6: Data Analysis and Interpretation of methods to test for inter-rater reliability exist,
Results—Calculation of the effect sizes and including percent agreement, the contingency
reporting them with a 95% confidence interval coefficient, Pearson’s r, the correlation coeffi-
(CI) are common practice with meta-analyses cient, the concordance correlation coefficient,
[25]. Numerous statistical programmes includ- and the most commonly used Cohen’s kappa for
ing the Cochrane-endorsed Review Manager two raters or Fleiss kappa for three or more rat-
programme (RevMan), MIX 2.0, and MetaStat ers [18]. Although heterogeneity can be judged
that conduct the meta-analysis processes are from graphical representations of data, such as
widely available. The results (effect sizes and CI looking at the error bars in forest plots, research-
intervals) should be reported graphically and ers should conduct a statistical test of heteroge-
quantitatively [25]. A common graphical repre- neity to address concerns of dissimilarities in
sentation of meta-analysis results is a forest plot study results within the meta-analysis. This test
(Fig. 37.4). In Fig. 37.4, the forest plot visually determines if the variation in the study results is
depicts each study as a square where the middle due to genuine measurable differences (hetero-
is the effect size (SMD) and each end point of geneity) or chance alone (homogeneity) but has
the corresponding line represents the upper and the inherent limitation of sensitivity to the num-
lower CI limits. The right portion of the plot (>0) ber of included trials [8]. The I2 statistic is com-
favours the control or comparator, whereas the monly used to quantitatively measure the
left (<0) favours the intervention [25]. The large variability between results (effect sizes of each
diamond at the bottom represents the pooled study). This assessment of consistency is critical
effect of all the individual studies. As the left as it directly relates to generalizability; the more
side of the graph favours the intervention, consistency in studies, the more generalizable it
researchers hope to see this diamond, or pooled is [18]. The statistic Cochran’s Q is commonly
effect, below <0 indicating efficacy of the inter- used to evaluate the null hypothesis of all
vention. Calculation of inter-rater agreement and included studies and evaluate similar effects [8].
tests of heterogeneity are also done in this phase. This test statistic is calculated by weighing the
Measurement of inter-­rater reliability is a key sums of squared deviated from the individual
component in the validity of a study as it repre- studies and the overall pooled result [14]. Finally,
sents how well the data collected are accurate a chi-­squared (χ2) distribution with k-1 degrees
representations of the variables of interest by of freedom is compared to the results from the Q
quantitatively measuring the extent to which test statistic to obtain P values [9].

Surgical treatment Conservative treatment

No. patients No. patients Favours Favours
Mean ± SD or knees Mean ± SD conservative surgical
Study or knees SMD (95% Cl)
Herrlin et al.38 93.5 ± 20 47 90 ± 11.9 49 0.21 (–0.19 to 0.61)
Katz et al.39 80.9 ± 17.8 161 80.7 ± 17.9 169 0.01 (–0.20 to 0.23)
Sihvonen et al.40 82.2 ± 16 70 83.4 ± 13.8 76 –0.08 (–0.40 to 0.24)
Vermesan et al.43 36.1 ± 3.6 60 34.7 ± 3.8 60 0.38 (0.01 to 0.74)
Yim et al.41 83.2 ± 12 50 84.3 ± 10.5 52 –0.10 (–0.49 to 0.29)

Overall 388 406 0.07 (–0.10 to 0.23)

Heterogeneity: l 2 = 20%
–1 –0.5 0 0.5 1
SMD (95% Cl)

Fig. 37.4 Meta-analysis forest plot

340 S. Akhter et al.

With interpreting the results, the researcher an electronic extraction form. The authors
should practise similar caution outlined in the employed statistical methods to report interob-
systematic review process. Summarizing server agreement and outcomes. All steps of the
findings, making evidence-based conclusions,
­ statistical methods are expertly reported and can
highlighting patient care implications, and sug- be readily reproduced. Test of heterogeneity
gesting future directions for research should be was also performed to determine if variability
included. was a result of chance or inter-study heteroge-
Clinical Vignette 2 [14] displays a meta-­ neity. Subgroup analyses were outlined a priori,
analysis on arthroscopic surgery for degenera- and sensitivity analyses were done to elucidate
tive tears of the meniscus. Authors systematically the consequences of missing data and studies at
searched three databases to ensure all relevant risk for bias. Figures of the study selection pro-
literature is captured. With a focus on random- cess and study descriptions were provided.
ized controlled trials, the search was conducted Results for the search, each individual outcome,
and studies screened and assessed for eligibility adverse events, and sensitivity analysis were
by two independent reviewers. To ensure a sys- reported and interpreted. Authors followed the
tematic and consistent process, the same review- PRISMA statement for reporting findings. The
ers independently conducted risk of bias authors concluded with limitations, implica-
assessments followed by data extraction using tions, and a conclusion.

Clinical Vignette 2: Meta-analysis

37 What Is the Difference Between a Systematic Review and a Meta-analysis? 341

• Lastly, researchers should appraise current

Fact Check reviews and meta-analyses for their methods
–– A meta-analysis involves a comprehen- as well as adhere to best practice guidelines to
sive and exhaustive review of relevant produce research that maximizes both internal
literature specific to a topic or research and external validity, resulting in clinically
question and can be viewed as an exten- relevant implications.
sion to a systematic review.
–– Meta-analyses employ statistical meth-
ods to quantitatively synthesize the References
results of pooled studies.
–– Performing a meta-analysis is more dif- 1. Aslam S, Emmanuel P. Formulating a research-
ficult and time-consuming than per- able question: a critical step for facilitating good
clinical research. Indian J Sex Transm Dis AIDS.
forming a systematic review but is of a 2010;31(1):47.
higher level of evidence. 2. Brighton B, Bhandari M, Tornetta P, Felson
–– The six steps in performing a compre- DT. Hierarchy of evidence: from case reports to ran-
hensive systematic review and meta- domized controlled trials. Clin Orthop Relat Res.
2003;413:19–24.
analysis are the same with one addition. 3. Burns PB, Rohrich RJ, Chung KC. The levels of evi-
Only step 6 involves an additional data dence and their role in evidence-based medicine. Plast
analysis via statistical software such as Reconstr Surg. 2011;128(1):305–10.
the Cochrane Collaboration-endorsed 4. Egger M, Davey-Smith G, Altman D. Systematic
reviews in health care: meta-analysis in context.
‘RevMan’. Somerset: Wiley; 2013.
5. GRADE Working Group. Grading quality of evi-
dence and strength of recommendations. BMJ.
2004;328(7454):1490.
6. Green S, Higgins JP. Preparing a cochrane review. In:
Take-Home Message Cochrane handbook for systematic reviews of inter-
• Both the systematic review and meta-analysis ventions; 2012. p. 11–30.
are considered the highest level of research 7. Guyatt GH, Sackett DL, Sinclair JC, Hayward R,
Cook DJ, Cook RJ. Users’ guides to the medical
evidence. literature. IX. A method for grading health care rec-
• A meta-analysis employs statistical methods ommendations. Evidence-Based Medicine Working
to quantitatively synthesize results of different Group. JAMA. 1995;274(22):1800–4.
studies and, when pooling high-quality ran- 8. Higgins JPT, Thompson SG, Deeks JJ, Altman
DG. Measuring inconsistency in meta-analyses. BMJ.
domized controlled trials, is considered the 2003;327(7414):557–60.
highest level of evidence. 9. Hopewell S, Mcdonald S, Clarke M, Egger M. Grey
• This design recognizes and adjusts results for literature in meta-analyses of randomized trials of
biases, increasing the internal and external health care interventions. Cochrane Database Syst
Rev. 2007;(2):MR000010.
validity of the study. 10. Jinha AE. Article 50 million: an estimate of the num-
• Conversely, a systematic review is a nonstatis- ber of scholarly articles in existence. Learned Publ.
tical, formal, and structured process governed 2010;23(3):258–63.
by carefully constructed stages which guide 11. Kagoma YK, Crowther MA, Douketis J, Bhandari M,
Eikelboom J, Lim W. Use of antifibrinolytic therapy
the cumulative search method, screening, to reduce transfusion in patients undergoing orthope-
reviewing, cataloguing, and reporting pro- dic surgery: a systematic review of randomized trials.
cesses of selected studies to answer a research Thromb Res. 2009;123(5):687–96.
question of interest. 12. Khan KS, Kunz R, Kleijnen J, Antes G. Five steps
to conducting a systematic review. J R Soc Med.
• Researchers must carefully consider which 2003;96(3):118–21.
method is indicated for the question of interest 13. Khan KS, Kunz R, Kleijnen J, Antes G. Systematic
as specific factors preclude specific designs, reviews to support evidence-based medicine: how
as seen with heterogeneity precluding the use to review and apply findings of healthcare research.
London: Royal Society of Medicine Press; 2003.
of a meta-analysis.
342 S. Akhter et al.

14. Khan M, Evaniew N, Bedi A, Ayeni OR, Bhandari ent reference values: workshop summary. Rockville:
M. Arthroscopic surgery for degenerative tears of the U.S. Dept. of Health and Human Services, Agency for
meniscus: a systematic review and meta-analysis. Can Healthcare Research and Quality; 2009.
Med Assoc J. 2014;186(14):1057–64. 22. Santos JRA. Cronbach’s alpha: a tool for assessing the
15. Lau J, Ioannidis JP, Schmid CH. Quantitative syn- reliability of scales. J Ext. 1999;37:2.
thesis in systematic reviews. Ann Intern Med. 23. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y,
1997;127(9):820. Chipponi J. Methodological index for non-­randomized
16. Liberati A, Al tman DG, Tetzlaff J, Murlow C, studies (MINORS): development and validation of a
Gøtzsche PC, Clarke M, et al. The PRISMA statement new instrument. ANZ J Surg. 2003;73(9):712–6.
for reporting systematic reviews and meta-­analyses 24. Torgerson C. Systematic reviews. London:
of studies that evaluate health care interventions: Continuum; 2003.
explanation and elaboration. Ann Intern Med. 25. Uman LS. Systematic reviews and meta-analyses. J
2009;151(4):W65–94. Can Acad Child Adolesc Psychiatry. 2011;20(1):57–9.
17. Matthew EF, Eleni EP, George AM, Georgios 26. Verhagen AP, Vet HCD, Bie RAD, Boers M, Brandt
P. Comparison of PubMed, Scopus, Web of Science, PAVD. The art of quality assessment of RCTs
and Google Scholar: strengths and weaknesses. Fed included in systematic reviews. J Clin Epidemiol.
Am Soc Exp Biol. 2015;20 Sep 2007. 2001;54(7):651–4.
18. Mchugh ML. Interrater reliability: the kappa statistic. 27. Weil RJ. The future of surgical research. PLoS Med.
Biochem Med. 2012;22(3):276–82. 2004;1(1):e13.
19. Moher D, Jadad AR, Nichol G, Penman M, Tugwell 28. Wright RW, Brand RA, Dunn W, Spindler KP. How
P, Walsh S. Assessing the quality of randomized con- to write a systematic review. Clin Orthop Relat Res.
trolled trials: an annotated bibliography of scales and 2007;455:23–9.
checklists. Control Clin Trials. 1995;16:62–73. 29. Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F,
20. Pae C-U. Why systematic review rather than narrative et al. The methodological quality assessment tools for
review? Psychiatry Investig. 2015;12(3):417. preclinical and clinical studies, systematic review and
21. Russell RM. Issues and challenges in conducting meta-analysis, and clinical practice guideline: a sys-
systematic reviews to support development of nutri- tematic review. J Evid Based Med. 2015;8(1):2–10.