Outcome and Impact Assessment

impact as
in International Development
Short presentation of the Zewo guidelines for projects and programmes

Zewo Foundation
Short presentation of the Zewo guidelines for projects and programmes 1
 The Zewo Foundation is the common aim of aid agen-
Swiss certification body for cies and donors. It is there-
non-profit organisations fore essential that organi-
that are involved in fund- sations think, plan and act
raising. It strives to pro- in a results-based manner
mote transparency and when planning, implemen-
integrity in fundraising and ting and assessing their
verifies that non-­profit projects and programmes.
organisations use the Assessing the effects they
funds entrusted to them in have achieved enables
This brochure gives a brief a conscientious manner. organisations to learn and
pre­sentation of the online Organi­sa­tions that comply improve. Aid agencies can
Zewo guide­lines for outcome with its requirements are concentrate their ­efforts
and impact assessment of awarded the Zewo seal where they are most
projects and programmes in of approval. needed and most ­effective.
international development These Zewo guidelines will
available at www.zewo.ch. Donations and public help aid agencies to ­adopt
© Published by Zewo Foundation, funding need to make a a results-based manage-
Zurich 2011 dif­ference: that is the ment approach.
CONTENTS

4 | Support for aid agencies

5 | 6 steps
6 | Outcome and impact assessment
8 | Step 1: Define the project objectives
10 | Step 2: Develop a results model

12 | Step 3: Plan impact assessment

14 | Step 4: Collect data

16 | Step 5: Evaluate the effects
18 | Step 6: Use the findings

20 | Resources and links

21 | Good practice

22 | Online guidelines
23 | Supporting institutions

Short presentation of the Zewo guidelines for projects and programmes 3

The guidelines help aid agencies to record the effects of their
projects and programmes. They can develop and implement an outcome and impact assessment system
that is tailored to their organisation’s needs.

The lessons learnt provide answers to the fundamental questions:

■■ Are we doing the right things ?

■■ Are we doing things properly ?
■■ How can we do things better ?

4 Outcome and impact assessment in international development

In
Seve-6 steps Outcome and impact assessment is built into the project cycle in 6 steps.
ral steps that are decisive need to be considered as early as in the planning stage. The various
stages may be repeated several times during the life of a project. The guidelines owe much to the Logical
Framework Approach, a results-based management model that is widely used in international development.
At the relevant points, alternative approaches are discussed, along with their corresponding methodologies.

The guidelines’ modular

structure makes the
introduction to the subject
of impact assessment
easier.

Short presentation of the Zewo guidelines for projects and programmes 5

Outcome and impact assessment aims primarily to record the
direct effects that aid agencies trigger off for the beneficiaries through their outputs, but it also measures
longer-term effects that extend beyond the target group.

Input Activities Output Outcome Impact

Expert Building Completed Children Improvement in

knowledge, schools schools, receive an education and
concepts, and training trained education prosperity
funds teachers teachers

Process evaluation Output Impact assessment

assessment

Planned activities Intended results

Standardised use of
terminology provides clarity.

6 Outcome and impact assessment in international development

Input
The financial, human, and material resour-
ces used for the development intervention. Output
The products, capital goods and servi-
Activities ces which result from a development inter­
Actions taken or work performed through vention; may also include changes resulting
which inputs, such as funds, technical assis- from the intervention which are relevant to
tance and other types of resources are mobi- the achievement of outcomes.
lized to produce specific outputs.
Outcome
The likely or achieved short-term and me­
dium-­term effects of an intervention’s out-
puts.

Impact
Positive and negative, primary and secon­dary
long-term effects produced by a ­development
intervention, directly or indirectly, intended
or unintended.

Source: OECD/DAC
Step 1: Define the project objectives
First of all, it must be clear what effect the project should have. There should also be analysis, together
with the target group, of their problems, the causes and their needs. The guidelines show what methodo-
logies exist, how to define outcome and impact objectives, and the areas that require particular attention.

The problem tree is

High child used to identify
mortality in problems and their
region x, y, z causes.

Children Children contract

contract malaria diarrhoea

Lack of effective Children are Mothers use Medical

medicine inadequately unclean drinking treatment is bad
protected from water
infection

Research Vaccine is Mothers know Clean water is Clinics are Lack of

into effective too expensive too little about too far away too far away medicine
medicine the links
halted

8 Outcome and impact assessment in international development

The project objective describes the desired direct, short – and medium – term effects that the project should
have on the target group. The project goal describes the long-term effects that the project is meant to set in
motion – or help to set in motion. It establishes the link between the project and the overarching goals, both
inside and outside the organisation. The guidelines show how to ensure that objectives are defined proper-
ly and aligned with each other, as this then makes it possible to translate them into measurable indicators.

Variant 1 ( e.g. EU )

Objective Indicator

Impact level Less children in x die from Child mortality in x < 5% ( by 2015 )
diarrhoea

Outcome level Improved access Walk to next well < 15 min for 90% of households in x (  by 2015  )
to drinking water

Output level Build wells 100 wells operational

The variant illustrated

above shows how objecti-
ves and indicators are de-
fined and used. The gui-
delines contain further
variants that are used in
practice, but these should
not be mixed up.

Short presentation of the Zewo guidelines for projects and programmes 9

Step 2: Develop a results model
If one is to measure and verify the effects of a project, it must first be clear how the effects are to be achie-
ved. The guidelines show how to find solutions, choose a strategy of intervention and develop a results model.

Child mortality
in regions
x, y and z falls

Less children Less children

contract malaria contract
diarrhoea

Effective Children are More mothers Better medical

medicine is better protected use clean care for children
available from infection drinking water

Research is Free vaccines Mothers more People have More children More children
re-started for children aware of better access to can be treated receive effective
the links clean drinking in time medicine
water

Health courses Build wells Install mobile Hand out The objective
chosen strategy for young health clinics medicine tree is used to
desired side effects mothers
develop different
no intervention solutions.

10 Outcome and impact assessment in international development

The Logic Model is a simple and commonly used way of displaying a linear chain of
cause and effect and of illustrating how a project will work. The guidelines show what it is suitable for and
what alternatives there are.

Example of a health course

Input Activities Output Outcome Impact

2 million CHF Provide Mothers More mothers Mothers

for education courses have attended understand increasingly
courses the links use clean
drinking water

Example of a simple
Logic Model.

Short presentation of the Zewo guidelines for projects and programmes 11

Step 3: Plan the outcome and impact
assessment
Even before the project is implemented, plans must be laid for how its effects are to be measured and
­assessed. The outcome and impact objectives must be converted into indicators, and the methods used to
collect and analyse the necessary data must be planned. The guidelines demonstrate how to define indi-
cators, which qualitative and quantitative methods exist, what kind of comparison is suitable for what, and
what aspects require particular attention when assessing outcomes and impact.

In a before-and-after
comparison with a con­-
trol group, the de­velop­-
­ment of the target
group is set against the
develop­ment of a con­-
trol group that has ­bene­-
fited from none of the
Target group ­project outputs. This
­excludes external ­in-
Control group fluences and makes it
­possible to ­observe what
would ­have taken place
­without the project.
Start Middle End
However, this approach
involves a great deal of
work and is metho­do­­-
lo­gically challenging. The
­guidelines also give
alternative ­approaches.

12 Outcome and impact assessment in international development

 There are warnings
N. B. about pitfalls, stumbling
It is frequently the case that, in practice, only the final situation of a project is described. blocks and known
Yet, for an impact assessment, a simple description of the target group with no link to the sources of error, and
the essential points are
objectives, the initial situation or a control group is not sufficient.
highlighted.

IMPORTANT
A good indicator ought to be SMART:
• Specific: the indicator must be unambiguous and clear.
• Measurable: the indicator must be measurable and the costs for measurements
appropriate.
• Achievable: the target value given by the indicator must be achievable.
• Relevant: the information provided by the indicator should be relevant for
the project manager.
• Time-bound: the indicator must show when the objective ought to be achieved.

Source  : European Commission, PCM Guidelines

Short presentation of the Zewo guidelines for projects and programmes 13

Step 4: Collect data
Ideally, the data needed for the outcome and impact assessment should be collected during project imple-
mentation. The guidelines show how to integrate outcome indicators into the project monitoring system,
what to pay attention to during data recording, and how to detect and correct deviations at an early stage.

Child health programme

Strategy of intervention Indicator Source 2009 2010 2011 2012
Impact Contributes to improved child Child mortality in Regions x, y National x:10% x:5%
health: child mortality falls and z falls from 10% to 2% statistics y:10% y:7%
z:10% z:12%

Outcome Mothers know about links Participants in the courses Video, analysis Good Satis-
can use the information by project factory
learned in a role-playing managers
game.

Ill children can be 95% of cases of children Case studies 89% 80%
successfully treated treated for diarrhoea are
successful.

Improved access to clean Walking time to nearest well Observation 50% 60%
drinking water < 15 minutes for 80% of
households

Output Courses 100 courses provided Project report 23 42

Cases treated 1,000 cases treated per year Treatment 955 1112
statistics

Wells 50 new wells in the region Project report 12

The logframe is a
Activites Provide courses
standardised table
Introduce mobile that summarises
health clinics
how the project is in-
Build wells tended to work as well
as its monitoring and
evaluation system.

14 Outcome and impact assessment in international development

Interim analysis of health course
Objective Indicator Target Perfor- Difference Comments Operative Consequences
mance measures at outcome level
O1 Use in role play Min. good Good – Analysis easy None

P1 N° of courses 1 (Pilot) 1 (Pilot) – None

P2 N° of 35 45 + 30% High demand Larger course Target group

participants groups ­larger than
assumed

Cost 10 000 15 000 + 5 000 Higher Check savings Implementati-

attendance, per course on might be less
higher costs day, apply ­efficient
for increased
budget

Deadline May 09 June 09 1 month Finding Need to train ­Outcome

late speaker additional objectives will
harder than speakers be achieved less
expected quickly than
­planned due to
need to train
speakers

Legend for colour code

Result Measure
Satisfactory Plan reinforcement and ensure continuous success
Take steps to correct This example
Critical
Keep under observation shows what
Still uncertain
None needed an interim
As planned
analysis might
look like.

Short presentation of the Zewo guidelines for projects and programmes 15

Step 5: Evaluate the effects
The guidelines help project managers to consolidate and interpret the qualitative and quantitative data. It
shows when effects can be attributed to a specific project as well as how to aggregate the effects of dif-
ferent projects.

How it is done

Activities Project managers or external experts make comparisons and find out the
project’s effect on the target group using the available data. This task should
be carried out according to standard evaluation practice. The findings are
generally recorded in writing.

Questions Coming up with answers to the following question forms the fifth step in an
impact assessment:
• Is all the necessary data available in a suitable format?
• What was the effect or change on the target group?
• What would have changed for the target group without the project?
• What are the reasons for any deviation from the project objectives?
• Which assumptions and hypotheses have proved true, and which
were false?
• What foreseen and unforeseen side effects were there?
• Is there a plausible case to be made that the project has contributed
to the overarching goals?
• Which effects can be clearly attributed to the project?
• Which recommendations are needed?

Results A report or a presentation has been made about the effects of the project
or programme.
The chapter summary gives
an overview for each step.

16 Outcome and impact assessment in international development

 Templates make work
easier for the users.

Template
for how to structure an impact assessment report
I Summary
II Basic principles
1. Rationale, purpose and objectives
2. Scope of the impact assessment
3. Questions for impact assessment
3.1 Question a
3.2 Question b
4. Context of the impact assessment
5. Team
III Approach
1. Discussion of methodology, sources of information
and data quality
2. Inclusion of relevant stakeholders
The outcome and impact assessment ­findings
IV Findings
1. Question a should always be reported in a suitable form,
1.1 Observations regardless of whether the findings are ex­pec­
1.2 Appraisal and conclusions ted or unexpected, negative or positive. This
2. Question b
2.1 Observations is often via a written report. The guidelines
2.2 Appraisal and conclusions show how these are generally structured and
V Overall conclusions and recommendations what other means of communication one might
­envisage.

Short presentation of the Zewo guidelines for projects and programmes 17

Step 6: Use the findings
The assessment’s findings can be used to build up organisational knowledge and to learn lessons for the
future, to steer the organisation’s activities towards results or to inform donors, partner organisations and
the target groups about the projects’ and programmes’ effects.

Learning

Steering

Legitimising

18 Outcome and impact assessment in international development

The guidelines show what is required if the learning process is to succeed, how to use the findings as a ­basis
for decision-making, and how to include them in the organisation’s yearly performance report according to
Swiss GAAP FER. There is a clear summary of the principles of good outcome and impact assessment, which
also shows how much impact ­assessment should cost.

A template and practical

examples give suggestions
as to how to include
the results of the impact
assessment in the
performance report.
(  Available in German
and French  )

Short presentation of the Zewo guidelines for projects and programmes 19

Resources and links
The guidelines give clear summaries of the main approaches and methodologies for impact assessment.
There is also a practical and well-structured list of links and downloads on the subject of outcome and
­impact assessment.

Approaches and methodologies

■■ Logical Framework Approach – is the most widespread model for results-based project planning in
­development circles.
■■ Outcome Mapping – is an alternative approach to developing a system to record the (qualitative) effects
of projects and programmes.
■■ Theory of Change – is an approach for results-based project planning and is based on a somewhat more
open results model than the Logical Framework Approach.
■■ Most Significant Change – is a very specific, qualitative and participatory technique for recording the
­effects of projects and programmes. It is based on the systematic analysis of individual experiences and
thus dispenses entirely with indicators and figures.
■■ MAPP (  Method for Impact Assessment of Programmes and Projects  ) – is a specific, participatory ­method
for recording the effects of projects and programmes. It is based on group discussions during which ­effects
and developments are analysed retrospectively following a set programme.
Links
■■ Overviews of different methodologies
■■ Handbooks on monitoring and evaluation
■■ Toolkits with ready-to-use tools for specific cases
■■ Field reports

20 Outcome and impact assessment in international development

Good Practice
There is no single impact assessment system. Every organisation must develop a plan that is adapted to its
own circumstances. In order for this to succeed the guidelines set out a few basic principles for ensuring
assessment quality and for publishing and communicating results.

The Zewo Foundation would like to promote widespread adoption of systematic impact assessment in
­practice and encourage aid agencies to develop and implement impact assessment systems that are tailored
to their needs as part of this good practice.

Short presentation of the Zewo guidelines for projects and programmes 21

Online guidelines
You can find the guidelines on-
line at www.zewo.ch/impact/en,
where they are also available as a
PDF file.

22 Outcome and impact assessment in international development

Supporting institutions
The Swiss Agency for Development and Cooperation ( SDC ) is the inter-
national development body within the Federal Department for ­Foreign
Affairs ( EDA ). It played an important part in developing and funding
­these guidelines.

The Swiss Evaluation Society ( SEVAL ) fosters the exchange of infor-

mation and experience in the field of evaluation. It actively engages in
­improving the quality of evaluation and its diffusion. SEVAL supports
the launch of these guidelines as a communications partner.

The ETH Zurich’s Centre for Development and Cooperation ( NADEL )

trains young and experienced professionals working in ­de­velopment co-
operation. In its courses NADEL teaches, among other things, methods
of results-based planning and impact analysis, as well as the Zewo gui-
delines for outcome and impact assessment.

We would like to thank the members of the working group – Bernard Du Pasquier ( HEKS ), Diether ­Grünenfelder
( EcoSolidar ), Maya Natarajan ( IAMANEH Switzerland ), Constanze Bunzemeier ( Enfants du Monde ),
­Peter Schmidt ( Helvetas ), Gerhard Siegfried ( SDC ) and Christian Varga ( Caritas Switzerland), as well as
Oliver Bieri and Stefan Rieder from Interface Policy studies Research Consulting.

Short presentation of the Zewo guidelines for projects and programmes 23

Zewo Foundation
Lägernstrasse 27
8037 Zurich, Switzerland
Telephone +41 (0)44 366 99 55
Fax +41 (0)44 366 99 50
[email protected]
www.zewo.ch