Chapter 2.

Social Experiments (Randomized Control

Trials and Natural Experiments)

Joan Llull

Quantitative Statistical Methods II

Barcelona School of Economics
 Randomized Experiments
In the treatment eect approach, a randomized eld trial is regarded as the ideal design.

Long history of randomized eld trials in social welfare in the U.S., beginning in the 1960s
(see Mott (2003) for a review).

Encouraged by U.S. Federal Government, eventually almost mandatory. Legislation intro-

duced in 1988.

Resistance from many states on ethical grounds (more so in other countries, where treat-
ment groups are often areas for treatment instead of individuals).

Sometimes experiments are provided by nature: natural experiments (e.g. John Snow
and the cholera case in SoHo).

Chapter 2. Social Experiments 2

 Random Assignment and Treatment Eects
In a controlled experiment, treatment status is randomly assigned by the re-
searcher, which by construction, ensures independence:
Y1i , Y0i ⊥
⊥ Di .

This eliminates the selection bias (see Chapter 1), and implies αT T = β , as:
E[Y0i |Di = 1] = E[Y0i |Di = 0] = E[Y0i ].

Also αAT E = αT T = β , as E[Y1i − Y0i |Di = 1] = E[Y1i − Y0i ].

Thus, the average treatment eect can be estimated by a simple linear regression
of the observed outcome Yi on the treatment dummy Di and a constant.

Chapter 2. Social Experiments 3

 Standard Errors and Inference
The standard default options for regression in statistical packages assume that resid-
uals Ui are homoskedastic. This implies:
Var(Yi |Di = 1) = Var(Yi |Di = 0) = Var(Ui ) = Var(Y0i ) = Var(Y1i ).

In the context of heterogeneous treatment eects, this is often violated.

Var(Y1i ) can be expressed as:

Var(Y1i ) = Var(Y0i ) + Var(Y1i − Y0i ) + 2 Cov(Y1i − Y0i , Y0i ),
which is obtained by noting that Var(Y1i ) = Var(Y0i + (Y1i − Y0i )).

Homogeneous treatment eects ⇒ Y1i − Y0i is a constant ⇒ Var(Y1i − Y0i ) =

Cov(Y1i − Y0i , Y0i ) = 0.

Heterogeneous treatment eects ⇒ not necessarily the case (e.g. eect of lottery
ticket on wealth).
Chapter 2. Social Experiments 4
 Standard Errors and Inference
Standard error of the ATE is the variance of the dierence in means:
S σT2 σ2
Var(β ) = Var(ȲT − ȲC ) = Var(ȲT ) + Var(ȲC ) = + C,
N1 N0
where σT2 and
2
σC are sample variances of the outcome computed on treated and control subsamples,
and N1 and N0 are the sizes of each subsample.

Alternatively, from the regression we can compute robust standard errors:

 
β̂0 1
Var = E[Xi Xi0 ]−1 E[Xi Xi0 Ûi2 ] E[Xi Xi0 ]−1 ,
β̂ N
where Xi ≡ (1, Di )0 . The sample analog is:
  N
!−1 N
! N
!−1
β̂0 1 1 X 0 1 X 1 X
Var
d = Xi Xi Xi Xi0 Ui2 Xi Xi0
β̂ N N i=1 N i=1 N i=1
 P 2 P 2

i:Di =0 Ûi i:Di =0 Ûi
N02
− N02
= .
 
Ûi2 2
Ûi2 
P P P
i:Di =0 i:Di =0 Ûi i:Di =1
− N02 N02
+ N12

Maybe observations are not independent ⇒ clustering (e.g. Progresa).

Chapter 2. Social Experiments 5
 Introduction of Additional Regressors
Additional regressors Wi are not needed for consistency as:
Cov(Wi , Di )
γ = 0.
Var(Di )

Yet, it can be interesting to include them for several reasons:

If they are relevant, they can increase precision (Frisch-Waugh Theorem).
Checking randomization: are there statistical dierence in these regressors
between treated and controls?
Used in the randomization (e.g. village-level randomization).

The last two lead to the context of conditional independence.

Chapter 2. Social Experiments 6
 Warning: Partial Complience

So far we have assumed perfect complience: everyone elected takes the treatment
and no control takes it.

Now: Di = 1{treatment taken} and Zi = 1{assigned to treatment}.

We may have Di = 0 and Zi = 1 (no-shows), and Di = 1 and Zi = 0 (cross-overs).

Now Y1i , Y0i ⊥ ⊥ Zi . The latter can be used in an IV fashion to

6 ⊥ Di but Y1i , Y0i ⊥
obtain αT T (Chapter 4), or compute an intention-to-treat eect.

Chapter 2. Social Experiments 7

 Warning: Longer Run Outcomes
National Supported Work program (NSW):
designed in the U.S. in the mid 1970s
training and job opportunities to disadvantaged workers
NSW guaranteed to treated participants 12 months of subsidized employment
(as trainees) in jobs with gradual increase in work standards.
experimental design on women who volunteered for training
Requirements: unemployed, a long-term AFDC recipient, and have no preschool
children
Participants were randomly assigned to treatment (275) and control groups (266)
in 1976-1977
Training in 1976, and then followed.
Ham and LaLonde (1996) analyze the eects of the program.
Chapter 2. Social Experiments 8
Chapter 2. Social Experiments 9
 Eects on Unemployment Rates
Thanks to randomization, comparison of employment rates for treated and control gives an
unbiased estimate of the eect of the program on employment at dierent horizons.

Initially, by construction there is a mechanical eect from the fact that treated women are
oered a subsidized job.

Compliance with the treatment is decreasing over time, as women can decide to drop from
the subsidized job.

The employment growth for controls is a reection of the program's eligibility criteria.

Importantly, after the program ends, a 9 percentage points dierence in employment

rates is sustained.

Chapter 2. Social Experiments 10

 Ham and LaLonde's Additional Point
But Ham and LaLonde (1996) make an important additional point: randomization does
not guarantee independence for any possible outcomes.
Two examples: wages and unemployment durations (hazards).
Eect of training program on employment rates of the treated ⇒ those who are working are
a selected sample.

Notation: Wi wages; Yi = 1 if employed; ηi = 1 skilled type.

Suppose:
P (Yi = 1|Di = 1, ηi = j) > P (Yi = 1|Di = 0, ηi = j), j = 0, 1
and:
P (Yi = 1|Di = 1, ηi = 0) P (Yi = 1|Di = 1, ηi = 1)
> .
P (Yi = 1|Di = 0, ηi = 0) P (Yi = 1|Di = 0, ηi = 1)
Chapter 2. Social Experiments 11
 This implies that the frequency of low skill will be greater in the group of employed
treatments than in the employed controls:
P (ηi = 0|Yi = 1, Di = 1) > P (ηi = 0|Yi = 1, Di = 0),

which is a way to say that ηi , which is unobserved, is not independent of Di given Yi = 1,

although, unconditionally, ηi ⊥ ⊥ Di .

Consider the conditional eects:

∆j ≡ E[W |Yi = 1, Di = 1, ηi = j] − E[Wi |Yi = 1, Di = 0, ηi = j], j = 0, 1

Our eect of interest is:

∆AT E = ∆0 P (ηi = 0) + ∆1 P (ηi = 1),

and comparison of average wages between treatments and controls is:

∆W = E[Wi |Yi = 1, Di = 1] − E[Wi |Yi = 1, Di = 0] < ∆AT E .

⇒ may not be possible to correctly measure the eect on wages.

Chapter 2. Social Experiments 12