School of Computer Science and Electronics Engineering, University of Essex

ILecture 8: Introduction to Experimental Design and

Causality
CE880: An Approachable Introduction to Data Science

Haider Raza
Tuesday, 07th March 2023

1
About Myself

I Name: Haider Raza

I Position: Senior Lecturer in AI
I Research interest: AI, Machine Learning, Data Science
I Contact: [email protected]
I Academic Support Hours: 11-12 on Tuesday via zoom. Zoom link is available
on Moodle
I Website: www.sagihaider.com

2
What we will be covering in this lecture

I Experimental design
I Randomization
I A/B testing
I Causal inference

3
Experimental Design
What is Experimental Design?

Experiments are used to study causal relationships. You manipulate one or more
independent variables and measure their effect on one or more dependent variables.

4
What is Experimental Design?

Experiments are used to study causal relationships. You manipulate one or more
independent variables and measure their effect on one or more dependent variables.
Experimental design means creating a set of procedures to systematically test a
hypothesis. A good experimental design requires a strong understanding of the
system you are studying.

4
What is Experimental Design?

Experiments are used to study causal relationships. You manipulate one or more
independent variables and measure their effect on one or more dependent variables.
Experimental design means creating a set of procedures to systematically test a
hypothesis. A good experimental design requires a strong understanding of the
system you are studying.
There are five key steps in designing an experiment:

I Consider your variables and how they are related

I Write a specific, testable hypothesis
I Design experimental treatments to manipulate your independent variable
I Assign subjects to groups, either between-subjects or within-subjects
I Plan how you will measure your dependent variable

4
Step 1: Define your variables

Example question 1: Phone use and sleep You want to know how phone use before
bedtime affects sleep patterns. Specifically, you ask how the number of minutes a
person uses their phone before sleep affects the number of hours they sleep.

5
Step 1: Define your variables

Example question 1: Phone use and sleep You want to know how phone use before
bedtime affects sleep patterns. Specifically, you ask how the number of minutes a
person uses their phone before sleep affects the number of hours they sleep.
To translate your research question into an experimental hypothesis, you need to
define the main variables and make predictions about how they are related.

5
Step 2: Write your hypothesis

You have a testable hypothesis that addresses your research question.

6
Step 3: Design your experimental treatments

First, you may need to decide how widely to vary your independent variable. Second,
you may need to choose how finely to vary your independent variable.
You can choose to treat phone use as:

I a categorical variable: either as binary (yes/no) or as levels of a factor (no

phone use, low phone use, high phone use).
I a continuous variable (minutes of phone use measured every night).

7
Step 4: Assign your subjects to treatment groups

I First, you need to consider the study size: how many individuals will be included
in the experiment? In general, the more subjects you include, the greater your
experiment’s statistical power, which determines how much confidence you can
have in your results.
I Second, you need to randomly assign your subjects to treatment group
(variable group/experimental group). Each group receives a different level of
the treatment (e.g. no phone use, low phone use, high phone use).
I Third, you should also include a control group, which receives no treatment.
The control group tells us what would have happened to your test subjects
without any experimental intervention.

8
Step 4: Assign your subjects to treatment groups

When assigning your subjects to groups, there are two main choices you need to make:

I A completely randomized design vs a randomized block design.

I A between-subjects design vs a within-subjects design.

9
Randomization

An experiment can be completely randomized or randomized within blocks:

I In a completely randomized design, every subject is assigned to a treatment

group at random.
I In a randomized block design, subjects are first grouped according to a
characteristic they share, and then randomly assigned to treatments within those
groups.

10
Between-subjects vs. within-subjects

I In a between-subjects design (also known as an independent measures design or

classic ANOVA design), individuals receive only one of the possible levels of an
experimental treatment.
I In a within-subjects design (also known as a repeated measures design), every
individual receives each of the experimental treatments consecutively, and their
responses to each treatment are measured.

11
Between-subjects vs. within-subjects

I In a between-subjects design (also known as an independent measures design or

classic ANOVA design), individuals receive only one of the possible levels of an
experimental treatment.
I In a within-subjects design (also known as a repeated measures design), every
individual receives each of the experimental treatments consecutively, and their
responses to each treatment are measured.

12
Step 5: Measure your dependent variable

Finally, you need to decide how you’ll collect data on your dependent variable
outcomes. You should aim for reliable and valid measurements that minimize bias or
error.
In your experiment about phone use and sleep, you could measure your dependent
variable in one of two ways:

I Ask participants to record what time they go to sleep and get up each day.
I Ask participants to wear a sleep tracker.

13
A/B Testing
A/B Testing

A/B testing in its simplest sense is an experiment on two variants to see which
performs better based on a given metric.
Example: two consumer groups are exposed to two different versions of the same
thing to see if there is a significant difference in metrics like sessions, click-through
rate, and/or conversions.

14
A/B Testing

Using the visual above as an example, we could randomly split our customer base into
two groups, a control group and a variant group. Then, we can expose our variant
group with a red website banner and see if we get a significant increase in conversions.
It’s important to note that all other variables need to be held constant when
performing an A/B test.

15
How to conduct a A/B test: Formulate your hypothesis

Before conducting an A/B testing, you want to state your null hypothesis and
alternative hypothesis:

I The null hypothesis is one that states that sample observations result purely
from chance. From an A/B test perspective, the null hypothesis states that there
is no difference between the control and variant group.
I The alternative hypothesis is one that states that sample observations are
influenced by some non-random cause. From an A/B test perspective, the
alternative hypothesis states that there is a difference between the control and
variant group.

16
How to conduct a A/B test: Formulate your hypothesis

There are two important concepts to consider in this step, random samplings and
sample size.

I Random sampling is a technique where each sample in a population has an equal

chance of being chosen. Random sampling is important in hypothesis testing
because it eliminates sampling bias, and it’s important to eliminate bias because
you want the results of your A/B test to be representative of the entire
population rather than the sample itself.
I Sample size is important that you determine the minimum sample size for your
A/B test prior to conducting it so that you can eliminate under coverage bias,
bias from sampling too few observations.

17
How to conduct a A/B test: compare the results

There are a few steps in determining this:

I First, you want to set your alpha, the probability of making a type 1 error.
Typically the alpha is set at 5% or 0.05
I Next, you want to determine the probability value (p-value) by first calculating
statistics
I Lastly, compare the p-value to the alpha. If the p-value is greater than the alpha,
do not reject the null!

18
Causality
What is causal inference?

Inferring the effects of any treatment/policy/intervention/etc.

Examples:

I Effect of treatment on a disease

I Effect of climate change policy on emissions
I Effect of social media on mental health
I Many more (effect of X on Y)

19
Motivating example: Simpson‘s paradox

20
Motivating example: Simpson‘s paradox

21
Motivating example: Simpson‘s paradox

22
Motivating example: Simpson‘s paradox

23
Motivating example: Simpson‘s paradox

24
Motivating example: Simpson‘s paradox

25
Motivating example: Simpson‘s paradox

26
Motivating example: Simpson‘s paradox

27
Motivating example: Simpson‘s paradox

Here in causal graph: Condition is a cause of treatment

28
Motivating example: Simpson‘s paradox

Here in causal graph: Condition is a cause of treatment

29
Motivating example: Simpson‘s paradox

Here in causal graph: Condition is a cause of treatment

30
Motivating example: Simpson‘s paradox

31
Motivating example: Simpson‘s paradox

Here in causal graph: Treatment is the cause of condition

32
Motivating example: Simpson‘s paradox

Here in causal graph: Treatment is the cause of condition

33
Motivating example: Simpson‘s paradox

Here in causal graph: Treatment is the cause of condition

34
Acknowledgment

A few slides are adapted from Brady Neal Causal Inference course.

35
36