Exercices

Question: The counterfactual is defined as…

A. The outcomes of people that did not get the program

B. The outcomes of people that wanted the program

C. What would have happened to people if everyone got the program

D. What would have happened to people in the absence of the program

Question: When is a good time to do a randomized evaluation?

A. After the program has begun and you are not expanding it elsewhere

B. When a positive impact has been proven using rigorous methodology

C. When you are planning to roll out a program with the intention of taking it
to scale

D. When a program is on a very small scale e.g one village with treatment and
one without
Question: Put these Components of program evaluation in the right order
- Program Theory Assessment
- Process Evaluation
- Needs Assessment
- Impact Evaluation
- Cost Effectiveness

Question: What we need for a cost effectiveness analysis?

A. Different target populations.
B. Standardize outcome measures.
C. Decision to make.
D. Only methodology.
Question: A treatment group is:
A. People who randomly assigned to get program
B. People who don’t assigned to get program

1
C. People who choose to get program
D. People who encourage others to get program
Question: A control group is:
A. People who randomly assigned to get program
B. People who don’t assigned to get program
C. People who choose to get program
D. People who encourage others to get program

Question: A process evaluation is:

A. Compare what happened with what would have happened
B. describe what happened
C. Need a control group
D. Need a comparison group
Question: The gold standard of impact evaluation is:
A. to make a large evaluation
B. to have an indicator to measure the output
C. to do a randomized evaluation
D. to make a target populations.

Question: When not to do a randomized evaluation?

A. When there is an important question you want/need to know the answer to;
B. When you have Time, expertise, and money to do it right;
C. After the program has already begun and you are not expanding elsewhere.
D. When there is an important question you want it

Question: Impact is defined as:

A. The improvement in outcomes for participants over the course of a
program;
B. The comparison of outcomes for people who receive a program versus
people who do not receive a program;
C. The outcomes for participants after the program has been implemented,
and the outcomes for the same participants at the same point in time had
the program not been implemented;
D. Something fundamentally unknowable.

Question:The counterfactual can be observed.

A. True
B. False

2
 Question: What does it mean to say that two groups are "systematically"
identical?

A. Each member in each group has a "twin" in the other group: that is, the
two groups have the same number of people, who exhibit the exact same
characteristics with each other;
B. That if we took an average on a number of indicators (height, income,
education level, etc.), the two groups would have the same averages;
C. Contingent on a certain variables, such as gender, individuals in the two
groups would have identical characteristics;
D. They would yield statistically significant results.
Question: Randomizing over schools is an example of _______
randomization.

A. Individual
B. Cluster 
C. A unit of
D. Well-designed

Question: What are some benefits of using a lottery (select all that apply)?

A. They are simple, common, and transparent;

B. They are always the fairest method of randomization;
C. They are often viewed as fair;
D. All of the above.
Case study
A project designed to improve health providers‘ knowledge, attitudes and
practices and to increase providers‘ awareness of violence against women as
a public health problem and a violation of human rights.
1) Construct a Results Chain
2) What are the main characteristic of good indicators
3) What are the indicators?

Question: Order these steps to construct the theory of change

- Community needs
- Output

3
 - Outcome
- Impact
- Problem or issue
- Assumption

Case study
There are increasing numbers of uninsured male workers, aged 40-55, in a
village of Nigeria due to local plant closings. As the bottom line of hospitals
shrink, the costs of uninsured care in local emergency rooms are negatively
impacting local health systems. To meet the human and financial needs of the
village, an accessible, free medical home must be created to offer medical care
and health education for village’s uninsured residents.
Question: match the elements in column A with those of B to obtain the theory
of change.

A B
Anticipate 25% decrease in the incidence of uninsured men Community
seeking care in the ER within 5 years. needs
Anticipate a 15% increase in males, aged 40-55, with a free Output
medical home
A free medical clinic should prove successful in this village, Outcome
because of its history of extraordinary volunteerism.
Village’s Medical Society officially encourages its 400 medical
professional members to volunteer 20 hours each year to help
medically underserved community residents. Village’s Nursing
Association is
also interested in collaborating with a free clinic. Memorial
Hospital has agreed to assist in planning and funding a free
clinic.
Providing appropriate, preventive primary care, free medical Impact
home and patient education to reduce the numbers of uninsured
males, aged 40-55
Memorial Hospital’s Annual Report states that 28% of Assumptions
uninsured male patients, aged 40-55, received emergency room
care in the previous year. For that, the Medical Society and
Memorial Hospital’s Task Force on the Uninsured is researching
ways to address the needs of the uninsured.

4
 Question : A community collaborative, including the local school district,
Extension, and the local UW-system campus has received a grant for a project
titled “A Day at the University.” The project is a post-secondary education day
for Hispanic students grades 7-8 held on the local UW campus. The school
district will release the students from school to attend the day long event which
will include workshops, a student panel, lunch, and an “informance.” Students
will be given an assignment to be shared in their schools reflecting the
knowledge gained during their “Day at the University.”Objectives for the day
are: the students will gain an understanding that college is a possibility for them
through advanced planning and wise choices; they will be able to explain basic
types of financial aid and how to qualify; they will know some key resources
available to help them as they move through high school; and they will meet
several successful Hispanic community leaders who are college graduates.This
grant is about 200.000 Dollars and will spend to cover the cost of facilities,
equipments, foods and so on. This program expected that the monetary benefits
(MB): income and productivity gains, etc. and non- monetary benefits (NMB):
improved health and safety, increased psychological well-being, empowerment,
self confident, quality of life, etc. for 5 years are presented in this table:
Year MB + NMB
0 0
1 20.000
2 25.000
3 25.000
4 35.000
5 40.000
Question :
With 3% Discount Rate, is this program worth running?
Question: Why a results chain ?
A. to provide stakeholders with “a logical, plausible outline” of how the
planned intervention can lead to the desired results;
B. To understand the different steps of impact evaluation;
C. To be sure about the efficiency of the program.

Question : Match theses words with their meanings

Imput Products or

5
 servicese

Activities Long-term effects on living

standards

Outputs Resources mobilized

Outcomes What the program does

Impact Short and medium effect on living

standards

Case study
In the early 2000s, the Population Council and Save the Children implemented
the Ishraq Program in rural Upper Egypt, establishing girl-friendly spaces to impart
life skills, build social networks, and foster leadership and self-confidence. As it
turned out, program benefits went beyond the targeted out-of-school adolescent
girls and extended to the parents of participants. Girls conveyed information from
their classes to their mothers, including health information. Additionally,
observing their daughters’ participation in public life had a strong impact on
mothers’ perceptions of their own place in the public sphere. Thanks to their
daughters’ involvement in Ishraq, mothers realized that they, too, had a right to
access public services.
Question: construct the Results Chain, indicators and assumptions
Match the question
Questions Types of evaluation
questions
Will we reach the goal of training 5,000 youth per year? Descriptive (D)
Is the program implementation delayed?
Normative (N)
How many youth participate (by age, sex, etc.)? Cause and Effect
(CaE)
How many program participants find employment within 3

6
months?
Are participants satisfied with the program?

As a result of the job training, do participants have

higher paying jobs than they otherwise would have?

Does the project contribute to reducing poverty in

the area?

Does participant income increase by 20%, as planned?

Do we spend as much as we have budgeted?

Question: Fill in the blanks with these words: No evaluation (monitoring

only) , Impact evaluation, Performance evaluation, Process evaluation,
Cost-effectiveness/Cost-benefit analysis

Type of evaluation questions to be answered

Descriptive only Normative Cause and

effect

…………. …………. ……..

………

………………………………….

Question:

1When an impact evaluation is usually justified? select all that apply

7
 A. when the program is strategically relevant and influential;
B. Innovative or untested;
C. Replicable;
D. When the program has to be stopped.
2 Experimental techniques, and quasi- experimental techniques are used :

A. To make a cost effectiveness analysis Cost-Effectivennd Cost-Benefit Analys

B. to make sure that the treatment and the comparison groups are as
similar as possible

C. Only an experimental techniques allow us to be sure that the treatment and

the comparison groups are as similar as possible

3 A program select beneficiaries by: (select all that apply)

A. Random assignment;

B. Eligibility ranking;

C. Selective targeting decision;

D. Only random assignment.

4 When Can we Use a Lottery Design? (select all that apply
A when the evaluation is planned in advance of implementation (prospective);
B when the program can serve only a fraction of eligible youth;
C When we have to measure short-, medium-, and long-term impacts of the
program

Case study
A youth employment program in Colombia that provided three months of in-
classroom training and three months of on-the-job training to young people aged
18–25 in the lowest socioeconomic strata of the population. The training
providers were instructed to recruit more candidates than they had room for in
their courses in case not everyone would eventually attend the training. The

8
recruitment of eligible youth is about 4500 and the selection of evaluation
sample is about 4350 young people and only 4030 will be treated .

Questions
1) Construct the Results Chain of this program

2) Which Impact Evaluation Method can we use in this case?

3) Present this evaluation design
Question:
1 The phase-in design is better suited than a lottery design in the case of :
(select all that apply)
A. A large programs;
B. A long run program;
C. Small or short-run programs.
Case study
The World Bank’s Economic Empowerment of Adolescent Girls program
in Liberia provides six months of training and six months of follow-up
activities with two different curricula: (1) skills training for wage
employment, combined with job placement assistance; and (2) business
development skills combined with links to microfinance. Mentorship is also
provided to all beneficiaries starting from the third month of training. To
evaluate its impacts, the World Bank chose an impact evaluation design
since this would allow for a quality randomized evaluation while also being
able to eventually serve all girls who have been promised training. The
evaluation took advantage of the natural rollout of the program and the
operational constraints that did not allow for training every- one at the same
time.
Questions
1) What is the suit impact evaluation design for this case?
2) What are the two main advantages and disadvantages of this evaluation
design

Question:

9
 1 Advantages of the discontinuity method are:
A) It takes advantage of existing targeting rules;
B) It provides unbiased estimates for participants near the cutoff;
C) Impact estimates are valid only for the margin near the cutoff
and cannot be generalized to people whose scores are
further away from the threshold;
D) It does not require randomization of any kind.
Case study

Informational campaigns for parents with school staff

Problems of truancy1 and discipline can contribute to many schoolchildren in industrialized

societies graduating from school without mastering basic skills. The school district of Creteil,
in France, is a densely populated area with high proportions of immigrants from mostly
Maghreb countries, and has very poor socioeconomic indicators. In such a setting, linguistic
and social barriers along with financial and logistical constraints can prevent parents from
paying closer attention to their children’s education.

There are high rates of truancy, violence and indiscipline among children and learning
outcomes are poor. Increasing parental involvement has been widely touted as a means of
overcoming difficulties in child learning and behavior. The program called “La mallette des
parents”2 was designed to foster parental involvement through a series of monthly debates
with the school staff on how to successfully manage the transition from primary school to
middle school. These debates provided parents of sixth graders with information on the
French school system and guidelines on how to assist children with homework. Can parental
involvement be used as a lever to improve educational outcomes in France? Does greater
engagement by parents improve discipline and behavior? Do classroom interactions also
result in positive effects for children whose parents don’t attend the meetings?

Questions:
1) Who is the target population? (0.5point)
2) What are the problems faced by the students?(0.5point)
3) What might be the hypotheses that can be tested? (1point)
4) Construct the Results Chain of this program (2.5 points)

1
absentéisme
2
Des outils et des ressources pour accompagner les enseignants dans la construction d'une relation de confiance
avec les parents.

10
 5) What are the main indicators related to this results – chains (1.5 points)

II Questions of multiple choices (4points)

1) A theory of change:
E. demonstrates the pathway of how to get from here to there (i.e. what is needed for
goals to be achieved);
F. is A long run program;
G. requires underlying assumptions to be detailed out in a way that they can be tested and
measured;
H. puts the emphasis first on what the organization wants to achieve rather than on what
the organization is doing;
I. All of the above.
2) An indicator is :
A. a specific, observable and measurable characteristic that can be used to show
changes or progress a program is making toward achieving a specific outcome;
B. There should be exactly one indicator for each outcome;
C. There should be at least one indicator for each outcome;
D. The indicator should be focused, clear and specific.
3) When should evaluations be conducted?
A. evaluations should be conducted at the beginning and end of an intervention process;
B. should be set after a certain number of trainings;
C. evaluations are usually conducted to answer key questions on the program‘s
performance and carried out when the staff or the donor wants to make key decisions
around the program – such as how to improve the program, which activities to continue
or discontinue and whether or not to scale up the program;
D. should be set after each aspect of planning and implementing the campaign.

4) Impact Evaluation can only answer

A. cause and effect questions ;
B. normative questions ;
C. descriptive questions ;
D. requires underlying assumptions to be detailed out in a way that they can be tested and
measured;
E. puts the emphasis first on what the organization wants to achieve rather than on what
the organization is doing;
F. All of the above.
5) When is a good time to do a randomized evaluation?
A. after the program has begun and you are not expanding it elsewhere;
B. when a positive impact has been proven using rigorous methodology;
C. when you are planning to roll out a program with the intention of taking it to scale;

11
 D. when a program is on a very small scale e.g one village with treatment and one
without.
6) What we need for a cost effectiveness analysis?
A. different target populations;
B. standardize outcome measures;
C. decision to make;
D. only methodology.
III Case study (4points)
In the early 2000s, the Population Council and Save the Children implemented the Ishraq
Program in rural Upper Egypt, establishing girl-friendly spaces to impact life skills, build social
networks, and foster leadership and self-confidence. As it turned out, program benefits went
beyond the targeted out-of-school adolescent girls and extended to the parents of participants.
Girls conveyed information from their classes to their mothers, including health information.
Additionally, observing their daughters’ participation in public life had a strong impact on
mothers’ perceptions of their own place in the public sphere. Thanks to their daughters’
involvement in Ishraq, mothers realized that they, too, had a right to access public services.

Question: construct the Results Chain, indicators and assumptions

IV Fill in the blanks with these words (3points): counterfactual, happened, estimate,
gold standard, absence, goal.
The main …… of impact evaluation is to …… what would have ………. in the
………of the program. A number of methods have been used to estimate the ………
and to understand why experimental designs are the ……….
V Questions (3points)
1) What does it mean to say that the groups are equivalent at the start of the
program?
2) What can happen over the course of the evaluation of an intervention (after conducting
the random assignment) to make the groups non-equivalent?
3) How does non-equivalence between groups (treated and control) at the end threaten3
the integrity of the experiment?

I Case study (14 points)

Case study: An awareness-raising Campaign around Domestic

Violence Legislation
To increase knowledge of the new domestic violence provisions among
community members (men and women) in the village of Sudan by 50% and to

3
Menace.

12
double the number of women and girls in this village who claim they would
report violence perpetrated against them, a local NGO spent an amount of
150.000 Dollars to train women and girls (about 100 girls and woman every
year) and cover the cost of facilities, equipments, foods and so on. The NGO
expected that the monetary benefits (MB): income and productivity gains, etc.)
and non- monetary benefits (NMB): improved health and safety, increased
psychological well-being, empowerment, self confident, quality of life, etc.
related to these 100 girls and women are presented in this table:

Year MB + NMB
0 0
1 12.000
2 25.000
3 25.000
4 35.000
5 40.000

Questions
1) Construct the Results Chain of this program
2) What are the Characteristics of good indicators
3) What can be the results indicators for this case study?
4) With 5% Discount Rate, is this program worth running?
Exercice : cost effectiveness
Assume that we have an investment in an immunization programme in the

health sector. The objective of the programme is to reduce infant deaths.

Therefore the number of infant deaths avoided is shown in the table below :

13
with the initial and recurrent costs as shown in the table below

Cost – effectiveness analysis is a method that consists of defining the

objectives of a project and choosing the solution that minimizes discounted
capital and recurrent costs for a given output or maximizes the output for a
given cost.

14