PD Collins 0805699c

How to read a paper

Critical Appraisal

Why read a paper?

To pass a coursework Knowledge It might benefit your patients by changing your clinical practice

Clinical Practice Meta Analysis Systematic Review Randomised Control Trial

Cohort Study

Case-Control Study

Case Series

Case Report

Why critically appraise?

All research is not equal!

Identify those papers which change your practice

First Impressions
Journal
Big Clinical 4: NEJM, Lancet, JAMA, BMJ Other journals may be very significant within their field- e.g. Stroke Impact Factor search

Authors
Person who did all the work comes first Professor/ head of unit comes last

Title should be succinct and specific

You should know what the experiment is May be up front about authors main conclusion

Abstract
A representative structured summary of the paper

Introduction
Should always contain:
Brief review of the literature

Identifies a gap in knowledge

Clearly and explicitly states:
an aim (what did they intend to do) a hypothesis (postulated results)

Materials and Methodology

The most important section and the section which must be most closely scrutinised

Should enable replication of the study without contacting the authors

Methodology Checklist
Inclusion and exclusion criteria Clear and standardised guidelines for data collection 1 + 2 end points prespecified Any subgroup analysis prespecified Appropriate statistical analysis prespecified Power Calculation performed Randomisation Blinding Stopping Criteria

Look:
at the inclusion and exclusion criteria

Think:
do these generate an appropriate, well defined study population to test the hypothesis?

These affect how generalisable the results are

Common to inflate the risk level of the trial population with additional inclusion criteria so there are more events to save money + time Less than 5% of asthma patients would meet the inclusion criteria of an asthma trial

Look:
What are the 1 and 2 end points? These should always be pre-specified Is this the most appropriate end point?
For some conditions no single end point is sufficient e.g. RA or MS

Think:

Is this a surrogate end point?

A surrogate endpoint is a laboratory result or clinical sign which is supposed to relate to a clinically meaningful endpoint
E.g. Tumour shrinkage vs survival Surrogate endpoints may make trials more economic But introduce additional uncertainty

Look:
How were participants recruited?

Think:
Does this bias the study population in some way?

Also about generalisability:

Ad placed in the newspaper 1 vs 2 care recruitment setting Single vs multiple centres

Look:
Is there an intervention?

Think:
Is it described in sufficient detail to allow exact replication?
drug dose delivery method timings titration

Too vague- have all the patients received the same intervention? Can the same intervention be achieved in routine clinical practice?

Look:
Is there a control? If not why not?

Think:
Is the control appropriate?
On the basis of current knowledge will those in the control group be harmed by their participation?

Look:
Is the intervention the only difference between the two groups?

Think:
Do the intervention group receive some additional care which is likely to influence the results?

e.g. extra investigations, extra contact with health professionals

Look:
Are patients randomised to groups?

Think:
Is it clear how randomisation was achieved and will this facilitate blinding?
Every 2nd patient? Envelopes Central computer algorithm

Is this the best way of randomising patients?

Randomisation -Remove selection bias -Distribute confounding factors equally -Enables blinding

Look:
Is it blinded? If not why not?

Single (subjects), double (subjects and clinicians) or triple blinded (subjects, clinicians and investigators)? Think:
How was blinding achieved? If I was a subject or investigator would I be likely to beat the blinding?

May be very difficult if there are consistent side effects or if additional investigations which it would not be ethical to administer to controls are required May require dummy investigations- is this ethical?

Look:
How are patients to be followed up? Are there serial assessments? How is their status determined at the end?

Think:

For serial measurements are there set times when they will occur? Are there any subjectivities to assessment? Multiple centres? Are there agreed standards?

Ideally assessments should be carried out by a single team to an agreed standard-e.g. examination of all imaging and bloods at one centre

Look:
Is there some form of safety committee?

Safety committees generally are a separate group from the investigators who assess results at agreed points for the two groups -Should this trial be stopped? -Ethical question Think:
Are they separate from the investigators? Are there clear pre-stated criteria for stopping the trial early?

Results
What did they find? How is it presented? Are the results: a) statistically significant b) clinically significant

Results
Look: at table 1 (usually) Think: has randomisation been successful?

Statistical Significance
Is there evidence of a genuine difference between groups? Values of biological variables commonly vary around a distribution- just because we get different numbers does not necessarily mean there is a difference between two groups (it may be accounted for by this natural variability)

Statistical inference can help us determine if there is a genuine difference

The criteria for statistical significance should be set before result collection or analysis -By convention if p<0.05 then significant -Essentially this means there is <5% chance that the difference between groups is due to chance -The lower the p value, the more robust the evidence of a genuine difference

Clinical Significance
Statistical significance is determined by your sample size, the variation between samples and the margin of difference In a large clinical trial there may be thousands of subjects per group meaning that even a very small margin of difference between groups will have a p<0.05 First determine if there is statistical significance (usually p<0.05)

Then look at the effect size

Consider the absolute difference as well as the relative No easy answers

Next look at the confidence interval- very simplistically this is a measure of how accurately we can estimate the true difference between groups The narrower the confidence interval, the more accurate our estimate of the true effects of this intervention Even if an intervention is shown to have a large, statistically significant effect on an outcome in a trial, if the confidence intervals are very wide we might only be able to gauge the effect of treatment very imprecisely and therefore it is unlikely to change clinical practice

Be wary of exciting findings on subgroup analysis if not prespecified Be devils advocate- are there alternative explanations aside from intervention?
Differences in management between groups Failures of randomisation Other effects of drugs- check differences in physiological stats at end of study as well as at start

Data Presentation
Is the data presented honestly?

Classic fudge: Using graphs with altered scales to make it appear like there is a larger difference between the groups

Discussion
Puts results in context Draws conclusions from this study -what has it shown -usually some attempt to present results as clinically relevant Where do we go from here? What is the next stage in research?

Be critical of their conclusions- are they congruous with yours? If you cant follow the discussion, reread the paper.

Overall impressions
Is it good science? Are the results clinically significant? Will this make me change my clinical practice?

Few tips
Easy to be overcritical- no trial is perfect but have the investigators done the best they could to minimise bias and find an answer? Worth noting (perhaps with a sneer) the funding body (e.g. Big Pharma) but not in and of itself an indication of bias Search for the article on PubMed- letters and authors replies can provide invaluable insight If in doubt it is worth a google- sometimes small things can unravel your trust in a paper

Useful Resources
Consort Statement SIGN: Critical Appraisal: notes and checklists Bandolier (Oxford)

Fleming TR, DeMets DL. Surrogate End Points in Clinical Trials: Are We Being Misled? Ann Intern Med. 1996;125:605-613
Darrell Huff How to lie with statistics

Materials from GEMS will be online over the next few days