Evaluation of Evidence

&
Chance, Bias and confounding

By: Habtamu Esubalew (BSc,

MPH)
Why do we need to evaluate evidence?
Threats to Validity
• Examination of the association between outcome of interest
and exposure or intervention requires accurate measurements
that are representative of the target population.
• A study’s validity, or how close the findings are to the TRUTH,
can be compromised by three things…
 CONFOUNDING
 BIAS
 CHANCE

• A study is considered valid only when the above three

alternative explanations have been eliminated or reduced
2
 Cont’d..
– Association is a statistical relationship b/t
events, characteristics, or other variables, called
FACTORS.
• change in one variable is responsible, directly
or indirectly, for an observed change in another
variable.

– Causation implies that without a given event the

other event can’t happen
• Epidemiologic studies cannot prove – but
supporting a causal association 3
 What are Factor
• An element, event or condition that may affect an individual's
health or brings a change in health condition.

There are two types

Positive or risk Negative or

factor conditions Protective factors are
associated with the those conditions that
increased probability lower the chances of a
of developing a person developing a
disease disease.
4
 Error

Non-systematic Systematic Error

Random Error
 Non-systematic Vs Systematic

Non-differential (random error) Differential error (Bias)

Influence exposure & outcome Influence exposure & the

equally outcome differently

-affects the Reliability or -affects the validity of a

precision of a study study

-dilutes the measure of -increase or decrease

association towards null, i.e, the strength of an
leads to underestimation. association
6
 Random Error Vs Bias

• Inaccuracy which is • Inaccuracy which is

similar in comparison different in size and
groups & can be produced direction in one group
than the other
by biological variations,
(differential error)
measurement variations,
recording error
• Effect is not the same
• Effect is the same for all for groups in the study
groups in the study
7
1. Random error
=non-differential
~ is an error occurring due to chance.
– i.e. it is divergence from the true value due to
chance.

~ Affects the precision of a study

 Dilutes the measure of association towards one side.
~ can never be eliminated,
– But can be minimized by
• careful measurement of exposure and outcome
status
• increasing sample size of study subjects.
 Chance
Chance must always be considered as an alternative
explanation to observed findings,

The sample selected may demonstrate an association by

chance not present in the larger community.

The role of chance should be quantify using tests of

statistical significance: by calculating a
– p-value and
– confidence intervals.
 Chance
Consider chance at two stages
1. Designing phase
Increase the sample size (the Power)

2. Analysis phase (Statistics)

A. Hypothesis testing (test of statistical
significance)
P-value’s role, (Chi square, Students-t-test, F-
test, etc.

B. Determination of confidence interval 10

 Statistical Significance

• A statistical test tells the amount that the result

obtained happened by chance alone
– is a function of the difference between the values observed &
values expected in the null hypothesis and the amount of
variability in the sample.

• Therefore, it has two components:

– a numerator (difference between observed and expected
values)

– a denominator (variability in the estimate)

 Meaning of a Finding

Small sample large sample

• “no significant difference” • no significant difference
tells us nothing [because tells us exposure &
of sampling variation] outcomes have no
association
• a “significant difference”
may also tell nothing & • a significant difference
may not be replicable reveals a true
difference, but the
findings may not be
clinically important
BIAS

13
 Systematic error = Bias

~ when a factor affect one group more than another

• Direction of effect is unpredictable hence is
difficult to prove

~ is when results differ from the true value in a

systematic manner.
It is undesirable
It can’t be ‘adjusted for’
– Little can be done to fix or remove a bias once it has
occurred
 Bias cont’d…
~ Systematic error (not random) in a study that leads
to an incorrect estimate (OR, RR, …) of the
association between exposure and disease
– can occur in the design, implementation, or analysis
stages of a study
Effect
– It can either underestimate or overestimate the
true measure of association
– affects the validity of a study
 Types of error

Random Error

Systematic Error

Selection bias Information bias

 Selection Bias
 refers to any error that arises in the process of
identifying & selecting the study subjects.
– Invitation
– Acceptance

 when there is a systematic difference b/t

characteristics of the people selected & those not
selected.

 Results in distortion in a measure of association

(RR, OR) due to selection or retention of persons in
your study
 cont’d…
• Selection bias is introduced while selecting
 source population or study subjects (eligibility
criteria)

– select based on the interest of the data collector/

investigator
– Study subjects does not represent the cases
originated in the population
– whenever conditions influence the subjects' choice
to participate.

• Selection bias is introduced in any of study designs

but more in case-control and surveys
 Types of selection Bias
• Volunteer bias
– non-response, Refusal, agreement to participate
– People who volunteer to participate or accept to
participate in a study, are different from those who
do not volunteer or refuse to participate or from
the general population.
– It may underestimate or overestimate the
association
 Cont’d…

Healthy-worker bias: common in occupational

studies
• People who are working have to be healthy enough
to do their job, so they tend to be more robust than
the general population, which necessarily includes
those who are disabled or seriously ill and hence
unable to work

• Using the general population as a comparison group

for an occupational cohort study lead to
underestimate the risk associated with an
occupation due to the fact that employed people
tend to be healthier than the general population
20
 Cont’d…
Non-response Bias:
 Volunteers may be different from those who are
enlisted
 This is due to differences in the characteristics b/t
the responders and non-responders to the study.
 Non-response reduces the effective sample size,
resulting in loss of precision of the survey
estimates.
 Rates of response in many studies may be related
to exposure status.

21
Selection Bias: solution
• Cross-sectional surveys
– Maximise response rates
– Improve sampling techniques (SRS, Systematic
S.)
– Gather information about ‘missing (non-
respondent)’ groups, don’t replace study
subjects
– Maximum care about inferences

– Careful selection of control groups, and should

have a similar criteria as in selection of cases
 Information Bias
• Systematic error introduced during measurement of
information (assessing & measuring of
information) on either exposure or outcome
variables
• Bias is introduced when measurement
(ascertainment) of exposure/ outcome or both is not
well done or inaccurate.
• If measurement bias occurs equally in groups being
compared, it could result in underestimation of the
true strength of relationship.
 Cont’d…

• Occurs after the subjects have entered the study

• Pertains to how the data are collected

• Results in subjects who are incorrectly classified as

either exposed or unexposed, or as diseased or not
diseased

• Can occur in every type of study design

24
 Sources of information bias:

• Subject variation
• Observer variation
• Deficiency of tools
• Technical errors in measurement
 Types of Information Bias
• Investigator/Interviewer: an interviewer’s
knowledge on the exposure & outcome may
influence the structure of questions & the manner of
presentation that influence the response
• Observer Bias: Observers may have
preconceived expectations of what they should find
in an examination
• Misclassification bias; Errors are made in
classifying either the disease or exposure status
 Cont’d…
• Recall bias: those with a particular outcome or
exposure remember events more clearly or amplify
their memories.
– Occurs as a result of difficulty to recall prior
exposures

• Social desirability bias- Occurs b/c subjects are

systematically more likely to provide a socially
acceptable response.
 Minimizing information Bias
• Choose appropriate study design
– Use the same method of assessment
• Use strict and randomized sampling procedure
• Follow strictly the activities in a protocol,.
• Use “Double-blinding“ procedure
• Choose "hard" (objective) rather than “soft” (subjective) outcomes
• Use multiple sources of information
• Choose and stick to standardized questioner & instrument
– Use closed-ended questions whenever possible
• Train and blind interviewers; and so on
• Pretest, or a pilot study
• Facilitate recall

28
Confounder

29
 Confounding
• is an apparent association b/t disease & exposure
caused by a third factor (Confounder)

 A confounder can:
create the appearance of an association when
the true association is null
create the appearance of a null association
when there is a true association
bias the measure of a true effect toward or
away from the null value
reverse the direction of a true association
 Characteristics of confounder

A confounder is

 risk factor for the outcome

 associated with the exposure (better: a direct or
indirect predictor of the exposure)
 not an intermediate on the causal pathway
between exposure and outcome
Illustration of Confounding

association of interest

Exposure Disease

association association

Confounding
variable
Associations of Smoking, Cancer and
Chat

association of interest

Chat chewing Cancer

association association

Smoking
 Effects of confounding

A) Confounding can change the apparent direction of

an effect

1. May give false positive result, i.e. wholly account for

an observed effect or totally or partially accounts for the
apparent effect

2. May give false negative result, i.e. Mask an

underlying true association

3. Reverse the actual direction of the association.

34
 cont’d…

B) Could be large & can lead to overestimation or

underestimation of an effect, depending on the
direction of the associations that the confounding
factor has with exposure and disease.

• Confounding must be dealt as quantitative problem;

the amount of confounding rather than mere
presence or absence is important to evaluate
Minimize introduction of confounding

In the Study Design:

In the Analysis:
 Randomization  Stratification

 Restriction  Multivariable Adjustment

 Matching
 Validity
Overall judgment of the study results
• Internal validity- is the extent to which the results of
a study reflect the true situation in the study sample
in the absence of any alternative explanations.
(chance, bias & confounding)
– the prime objective of study design, implementation,
analysis and interpretation is to maximize the internal
validity of a study.
• External validity -is about generalizability
– are the results of a study applicable to populations other
than the study population

37
Judgment of Causality

38
 Establishing
Could it be due
to Bias?
Association

No

Could it be due to
Confounding?
?
Observed

Could it be due to
Chance?
?
Could it be Cause

Apply criteria & make judgment of causality 39

Criteria for Causal Association
• Hill’s Criteria (1965)
1.Strength
2.Consistency
3.Specificity
4.Temporality
5.Biological gradient
6.Plausibility
7.Coherence
8.Experiment
9.Analogy
 1. Temporal Relationship
• Most important criterion that must always be met
– Exposure precedes disease development with
adequate elapsed time

2. Strength of the Association

The stronger the association the more likely that it is a
causal. strong --- if it is far from unit.
strong RR < 0.5 & >2
weak RR > 0.5
& <2
0.5 2
strong weak strong
>0.5 RR<2
 3. Biological Plausibility

• A relationship b/t a supposed cause & an

outcome — that is consistent with existing
biological and medical knowledge.
 4. Dose-Response Relationship
• If risk increases with increasing exposure, it supports the
notion of a causal association
Death rate Risk Ratio
per 100,000 pers-yr
Non-smokers 7 1.00
1-14 cigarettes/ day 46 6.57
15-24 cigarettes/ day 61 9.71
25+ cigarettes/ day 104 14.86

However, the absence of dose-response does not preclude causal association

 There is almost always a dose below which no response occurs or can
be measured
 There is also a dose above which any further increases in the dose will
not result in any increased effect
 5. Effect of Removal of Exposure on the
Outcome-Reversibility

• Similar to the dose-response relationship, the

presence of this criterion supports the notion of
causal association
– However, the absence does not preclude it

• Example: after quitting smoking, the amount of

specific-DNA adducts decreases in blood
 6. Replication of the Findings

• It is supportive of causal association if the same

finding can be replicated in different populations
and/or by using various study designs

7. Extent to Which Alternate Explanations Have

Been Considered in the Study

• Third factors
• Bias
• chance
 8. Specificity of the Association

• Specificity of the association suggests that one

exposure is specific to one disease
– This criterion is not applicable to all exposure-
disease associations b/c a disease may be caused
by several exposures, & an exposure may cause
several diseases
• An exposure is likely to have a harmful effect on a
specific mechanism (at a cellular or molecular level) that
may then lead to one or more diseases

– An exposure, such as cigarette smoking, is

comprised of many smaller chemical components
 9. Consistency with Other Knowledge

• In vitro studies
• Animal studies
• Other studies such as ecological studies, cross-
sectional studies
• Other types of data such as sales data, time
trend
• The same association should be demonstrated
by other studies both with different methods,
settings and different investigators.
 10. Study design

It is most important to consider the design.

Type of study Ability to give

evidence
Randomized controlled Most Stronger
trial W
E
Cohort study Stronger A
Case-control study Moderate K
N
Cross-sectional studies weak E
S
Ecological studies More weaker S
48
 Evidence of a causal relationship

• Major category
– Temporal
– Biological plausibility
– Consistency and replications
– Alternative explanations
– Study design
• Minor category
– Dose-response
– Strength of association
– Cessation effects
THANK YOU