Having collaborated with others to come up with a solution and design a strategy

for implementation, we still need to work out how to evaluate our own project.

The evaluation field is vast, with countless and diverse methods.

In the next two modules you will get introduced to a few innovative approaches to
help you:

1. run your own experiments

2. set up your own pilots
3. use social auditing to engage others in measuring what works

Randomized Controlled Trials (RCTs) can help to test whether small changes designed
to influence human behavior can make a costly intervention more effective. We can
try to implement our own RCTs as a way to generate evidence in real time about
whether the intervention we plan to pursue works in practice.

This often involves breaking a project into its key components, testing what can
make the difference, and attributing progress – or lack of it – to the relevant
factors.

The UK Behavioural Insights Team has conducted more than 400 RCTs.

To test a solution in the field, it recommends a nine-step process divided into

three parts – test, learn, adapt. This framework provides a useful summary of how
an RCT works.

In order to test, you identify two or more policy or service interventions to

compare.

You determine the outcome that the policy is intended to influence and how the
trial will measure it.

Then you will want to decide on the randomization unit.

For example, we might be dividing individuals into two groups, or we might be

randomizing at the level of the institution and dividing a cohort of schools into
groups.

We need to judge how big our groups need to be to get meaningful results. Next we
need to randomly assign our test participants into the experimental or control
group, and give the experimental group the intervention.

Then we can observe, and determine whether the trial has yielded any insight about
the success of the intervention.

Done right, you will be sure to comply with ethical principles and rules governing
human subjects research, which you can learn more about online or by talking to or
collaborating with a university-based professional.

Good RCTs demand a large enough sample size and the ability to randomly select
subjects from the sample. Both are needed to ensure that the populations in each
group are relatively similar and that it is reasonable to infer causality from the
intervention.

When those conditions can be satisfied, some assert that RCTs can be run quickly
and relatively inexpensively. Ideas42, a nonprofit consultancy, estimates the cost
of RCTs at a few hundred dollars.

Others are more critical, pointing out that embracing experimental methods like
RCTs demands significant investment of researcher time, talent and attention. A
study must be conceptualized and designed, groups have to be sorted, data gathered,
and permission obtained from relevant agencies and organizations.

In field experiments, it can be challenging to be sure the right intervention is

delivered to the right people and the outcomes are measured correctly. Fortunately,
digital technology can help make it easier to more efficiently check on this kind
of progress.

RCTs are powerful tools, but they have limits. When the intervention being tested
is a simple behavioral prompt, for example, RCTs can identify whether that specific
intervention works. But knowing what worked in a specific instance does not allow
us to generalize, or to answer important questions about the relevance of the
solution.

As Princeton Economist Angus Deaton writes:

Randomized controlled .. have no unique advantages or disadvantages over other

empirical methods in economics. They do not simplify inference, nor can an RCT
establish causality.….The most troubling questions concern ethics, especially when
very poor people are experimented on. Finding out what works, even if such a thing
is possible, is in itself a deeply inadequate basis for policy

Designing other kinds of empirical experiments without a control group can be just
as meaningful and efficient, if not more so.

While RCTs have surged in popularity only in recent decades, there is a

longstanding tradition of empirical study of natural data obtained from
observations.

We can return to this approach to measure impact when RCTs do not work.
Difference-in-difference studies measure the difference between two naturally
occurring groups. For example, the City of Chicago imposed a plastic bag tax in
order to deter single-use plastic. Because everyone in the city was subject to the
tax, an RCT was not possible. Instead, the team conducted a natural experiment,
comparing bag use before and after the tax went into effect in Chicago, as well as
in suburbs not subject to a bag tax. They were able to report that the intervention
reduced disposable bag use by 42 per cent.

The 19th century English physician John Snow’s famous study revealing that cholera
is transmitted through the water supply rather than through the air is an example
of difference-in-difference, quasi-experimental evaluation design.

At the time, households in London were provided with water from one of two
companies. One year, before the cholera outbreak of 1853, one of those companies
changed the source of its water to a cleaner supply from outside the city. People
drinking the cleaner water died at a rate of only 37 per 10,000 whereas people
drinking from the polluted Thames died at 315 per 10,000.

This natural experiment, involving diverse people divided into two large enough
groups, made it possible for him to draw a statistically significant comparison.

At the height of the California energy crisis at the turn of the millennium, P.
Wesley Schultz and colleagues wanted to measure the impact on energy consumption of
sending people messages that signaled to them where they stood relative to their
neighbors. In the experiment, households were given weekly feedback about the
amount of electricity they used compared with similar households in their
neighborhood.

They reasoned correctly that the messages would have different effects for those
who were above-average energy users and those who were light users.

While the heavy users would use less, the light users would use more (a boomerang
effect), with no net benefits in energy saved.

The researchers added one additional condition. Light users receiving a smiley face
emoticon did not suffer the boomerang effect, illuminating a novel strategy for
reducing energy consumption, which has since become mainstream practice by
utilities across the country. Trials of the strategy in 100+ subsequent deployments
over ten years led to more than $1 billion saved in household energy costs and a
reduction of
nearly 13 billion pounds of CO2 emissions

Instead of a simple experiment with one intervention and a control group, this
design used two interventions and no control group, simply studying the behavior of
participants before and after the intervention.

With big data becoming much more prevalent, and with easier ways to gather and
analyze large quantities of data with machine learning, we are able to conduct
statistically relevant empirical experiments without the cost or complexity of an
RCT.