BENEFITS AND LIMITS OF
RANDOMIZATION
2.4
Tailoring the evaluation to the question
Advantage: answer the specific question well
We design our evaluation to answer exactly the question we want to
answer,
we choose the place and time
Disadvantage: one evaluation only answers the specific
question(s) it was designed to answer
One treatment and comparison group, creates one difference so
usually answers only one question
General vs. specific research
Indonesia family life survey
Survey on education, health, assets. Representative of 83% of Indonesia,
4 waves since every 1993
Use to assess shocks or programs where happens to be some quasi
randomness in implementation
Used for evaluating school building program, (Duflo, 2000)
Community monitoring evaluation in Indonesia
Communities randomized to receive either more external or more local
monitoring of road projects
Know from start we can compare these two approaches to monitoring
Collect exactly the data needed to monitor corruption by digging up
random sample of road and weigh materials used
Few assumptions, transparent findings
We know there is no selection bias,
only systematic difference between treatment and comparison groups
is the program
Fewer assumptions than for quasi randomized
that we have controlled for all differences
that the formula for administering the program cut off was strictly
applied
Direct comparison between two groups gives us our basic
result
less complex analysis helps transparency of findings
Prospective evaluation
What is a prospective evaluation?
Evaluation designed in advance
Advantages
Collect specific data
Collaborative design and evaluation
Disadvantages?
Long term results emerge in the long run
Q: What approaches could give us long run results in the short run?
When is a randomized evaluation not useful?
When outcomes can only be measured at a very high level
Fixed vs. floating exchange rates
Freedom of the press
When general equilibrium effects are
important
Outcome (like price) is determined by the aggregate of hundreds or thousands
of interactions
If we change some interactions but not others, we can only observe the
aggregate effect
Only possible to study when markets are somewhat separate
Example: job counseling in France
Unemployed youth were given job counseling
Those given counseling found jobs sooner than those not
given counseling
But what was the overall effect on youth employment: did the
counseled youth just displace other youth?
Randomized how many youth counseled by city
Where more youth counseled, uncounseled did worse, i.e.
displacement was a real problem
Crepon et al 2013
Ethical considerations: background
Belmont principles (and equivalent in other countries)
establish ethical rules for research. Include 3 key principles:
Respect for persons: participants should be informed of risks and
given a choice about participation
Benefice: the risks of research should be carefully weighed against the
benefits. Risks should be minimized
Justice: the people (and the type of people) who take the risks should
be those who benefit
Implications: approval for study
Many universities have Institutional Review Boards who
review all human subject research of students, faculty and
staff
May also require approvals from review board where study
takes place
Approval needed before study starts
Good to get general approval at conceptual stage to cover piloting
Provide detailed surveys and final numbers of subjects covered before
study launch
Approvals usually need to be updated every year with updates of people
reached and any adverse consequences
Respect for persons
Informed consent must be gained from participants in the
study
Waivers given if minimal risk and cost of collecting consent is high,
e.g., knowing they are part of study will change behavior
substantially
Particular care needed for those who might not be able to
judge risks well or find it hard to say no (e.g. children and
prisoners)
Do you need informed consent for all those in treatment
communities when not everyone is surveyed?
Do people opt into the program or is entire community impacted?
Sometimes ask consent at community meeting
Data that could identify an individual must be kept confidential
As early as possible take out information that could identify
individuals
Use deidentified data for analysis and publication
Justice provision
Test questions of relevance to those involved in the study
Unethical to test a drug on prisoners and only sell drug to
rich people
Best if participants themselves gain from findings
E.g. the program found to work, scaled up in study location
Not always possible
Ethical to test program designed to help poor Ethiopian
farmers on poor Ethiopian farmers
Benefice principle
Potential benefits must outweigh potential harm
Researcher should seek to minimize risk
Is researcher responsible for risk of program or only for the risk
of harm associated with the evaluation?
Program may have risks but that does not make evaluating it
unethical
Are program participants informed of program risks?
Would the program have gone ahead anyway?
Researchers should seek to minimize risk in program they evaluate but
important to evaluate risky programs that are going ahead anyway
Angrist, 1990 evaluated impact of draft lottery for Vietnam War
Risk of harm from evaluating
Do fewer people receive the program because of the evaluation?
Ethical to reduce coverage if benefits of program unclear and evaluation
can improve effectiveness of program
Do different people receive the program?
Is the program less well targeted than it would have been without the
evaluation?
Cost of less good targeting needs to be offset against gains from
evaluation
Will evaluation help improve knowledge of who to target?
Risk of confidential data becoming public
See respect for persons
What are the benefits?
If we answer an important question well, this can have big
benefits to society and those studied
If we find harm, program can be shut down
If we find benefit, program can be extended
Learn lessons potentially relevant to other situations
The ability of randomized evaluations to learn about causality
is relevant for ethics
The better we answer the question the higher the benefit, key
component of ethics considerations
