Scribd
Upload
53 views18 pages

N. Course Summary - Final

This document summarizes the key concepts and topics covered during a course on evaluating policies and programs for maximum impact. Over three days, lectures covered impact evaluation methods like randomized evaluations and quasi-experimental designs, how to design and assess evaluations, and using evidence to inform policy. Case studies provided opportunities to apply concepts in small groups. The overall messages were that high-quality evidence is needed to ensure effective policies, randomized evaluations are most reliable when possible, and process and cost evaluations are also important to fully understand impact.

Uploaded by

DetteDeCastro
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
53 views18 pages

N. Course Summary - Final

This document summarizes the key concepts and topics covered during a course on evaluating policies and programs for maximum impact. Over three days, lectures covered impact evaluation methods like randomized evaluations and quasi-experimental designs, how to design and assess evaluations, and using evidence to inform policy. Case studies provided opportunities to apply concepts in small groups. The overall messages were that high-quality evidence is needed to ensure effective policies, randomized evaluations are most reliable when possible, and process and cost evaluations are also important to fully understand impact.

Uploaded by

DetteDeCastro
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

Reflection and wrap-up

Deanna Ford
Associate Director Evidence for Policy Design Harvard University
August 29, 2012

Resources
Glossary of terms Guide for assessing quality of randomized evaluation All course materials will be available on ADB/IPA course website
Lectures Case studies and teaching notes

TAs/organizers who can point you to additional resources

Day 1, Lecture 1
Title: Evaluating for Policy Design Lecturer: Deanna Ford Key concepts:
Impact is about finding the cause and effect relationship between a program and outcomes Evaluation can be used to inform the design of policy to achieve greater impacts High quality evidence is important to ensure that policies and programs are effective

Day 1, Lecture 2
Title: Process Evaluation Lecturer: Sharon Barnhardt Key concepts:
Key components of a needs assessment How to formulate and evaluate a program theory Key components of process evaluation Importance of process evaluation as prior step and key component of an impact evaluation

Day 1, Lecture 3
Title: How to Evaluate Impact Lecturer: Sharon Barnhardt Key concepts:
Impact is the difference between what happened with the program and what would have happened without the program Measuring impact is all about finding a convincing representation of the counterfactual Randomization is often the easiest way to create a reliable counterfactual

Day 2, Lecture 1
Title: Randomization Why and When Lecturer: Rema Hanna Key concepts:
Randomization provides best estimate of counterfactual and the true program effect by creating two groups that can be considered the same in every way except for receiving the intervention A large sample size is necessary to have similar treatment and control groups

Day 2, Lecture 2
Title: Randomization How Lecturer: Aishwarya Ratan Key concepts:
Various options in how to randomize program areas (basic, phase-in, encouragement) Importance of selecting the correct unit of randomization Threats such as attrition and spillovers need to be identified so that they can be dealt with

Day 2, Lecture 3
Title: Testing program design Lecturer: David Yanagizawa-Drott Key concepts:
Randomization can be used to test the design of a policy/program to provide evidence on: Cost-benefit of varying levels of program intensity Value of combining or adding program components Answers to key questions on how to solve a particular policy problem

Day 3, Lecture 1
Title: Quasi-experimental/methods Lecturer: Rema Hanna Key concepts:
a) b) c) Regression Discontinuity Matching Difference-in-Differences Strengths Threats Validity Tests

Day 3, Lecture 2
Title: How to evaluate an evaluation Lecturer: David Yanagizawa-Drott Key concepts:
To evaluate an evaluation, critically identify and assess the assumptions for the comparison group to accurately represent the counterfactual Having a comparison group that does not accurately represent the counterfactual will lead to false conclusions about program impact Across impact evaluation methods, assumptions are most easily assessed and most likely to hold in a randomized evaluation

Day 3, Lecture 3
Title: Cost-Effectiveness Analysis Lecturer: Shawn Powers Key concepts:
CEA Provides a simple basis of comparison combining two big things implementers care about: cost and impact Getting the right cost data is the biggest challengeeven harder to do retrospectively Many ways to implement CEA, but key is consistency in your assumptions

Day 3, Lecture 3
Title: From Research to Practice: Applying Evidence Across Country Contexts Lecturer: Annie Duflo Key concepts: To determine if you can apply the results of an evaluation, seek to understand the mechanisms of impact to see if the policy will be effective in the new area

Day 1, Case 1
Title: Grace Period Intervention Group work Key concepts:
Defining outcomes of interest Formulating program theory Identifying assumptions behind the program theory How to select good indicators

Day 1, Case 2
Title: Savings Groups Group work Key concepts:
How to identify the comparison group to measure program impact How to determine whether the comparison group is a good representation of the counterfactual To learn how to identify weaknesses in difference impact evaluation methods

Day 2, Case 3
Title: Extra-Teacher Program Group work Key concepts:
How to select the unit of randomization How to answer different research/policy questions with a randomized evaluation

Day 2, Case 4a
Title: Millennium Villages Group work Key concepts:
How to assess an evaluation

Day 2, Case 4b
Title: Deworming Group work Key concepts:
How to assess an evaluation

Key take-away points


A high-quality impact evaluation requires a well thought out program theory and process evaluation An impact evaluation with a weak comparison group will often give you the wrong answer Randomization is typically the most effective way to ensure you have a reliable control group RCTs allow you to confidently test policy design and cost-effectiveness in order to increase impact

You might also like