Outcome and Impact Assessment
impact as
in International Development
Short presentation of the Zewo guidelines for projects and programmes
Zewo Foundation
Short presentation of the Zewo guidelines for projects and programmes 1
The Zewo Foundation is the common aim of aid agen-
Swiss certification body for cies and donors. It is there-
non-profit organisations fore essential that organi-
that are involved in fund- sations think, plan and act
raising. It strives to pro- in a results-based manner
mote transparency and when planning, implemen-
integrity in fundraising and ting and assessing their
verifies that non-profit projects and programmes.
organisations use the Assessing the effects they
funds entrusted to them in have achieved enables
This brochure gives a brief a conscientious manner. organisations to learn and
presentation of the online Organisations that comply improve. Aid agencies can
Zewo guidelines for outcome with its requirements are concentrate their efforts
and impact assessment of awarded the Zewo seal where they are most
projects and programmes in of approval. needed and most effective.
international development These Zewo guidelines will
available at www.zewo.ch. Donations and public help aid agencies to adopt
© Published by Zewo Foundation, funding need to make a a results-based manage-
Zurich 2011 difference: that is the ment approach.
CONTENTS
4 | Support for aid agencies
5 | 6 steps
6 | Outcome and impact assessment
8 | Step 1: Define the project objectives
10 | Step 2: Develop a results model
12 | Step 3: Plan impact assessment
14 | Step 4: Collect data
16 | Step 5: Evaluate the effects
18 | Step 6: Use the findings
20 | Resources and links
21 | Good practice
22 | Online guidelines
23 | Supporting institutions
Short presentation of the Zewo guidelines for projects and programmes 3
The guidelines help aid agencies to record the effects of their
projects and programmes. They can develop and implement an outcome and impact assessment system
that is tailored to their organisation’s needs.
The lessons learnt provide answers to the fundamental questions:
■■ Are we doing the right things ?
■■ Are we doing things properly ?
■■ How can we do things better ?
4 Outcome and impact assessment in international development
In
Seve-6 steps Outcome and impact assessment is built into the project cycle in 6 steps.
ral steps that are decisive need to be considered as early as in the planning stage. The various
stages may be repeated several times during the life of a project. The guidelines owe much to the Logical
Framework Approach, a results-based management model that is widely used in international development.
At the relevant points, alternative approaches are discussed, along with their corresponding methodologies.
The guidelines’ modular
structure makes the
introduction to the subject
of impact assessment
easier.
Short presentation of the Zewo guidelines for projects and programmes 5
Outcome and impact assessment aims primarily to record the
direct effects that aid agencies trigger off for the beneficiaries through their outputs, but it also measures
longer-term effects that extend beyond the target group.
Input Activities Output Outcome Impact
Expert Building Completed Children Improvement in
knowledge, schools schools, receive an education and
concepts, and training trained education prosperity
funds teachers teachers
Process evaluation Output Impact assessment
assessment
Planned activities Intended results
Standardised use of
terminology provides clarity.
6 Outcome and impact assessment in international development
Input
The financial, human, and material resour-
ces used for the development intervention. Output
The products, capital goods and servi-
Activities ces which result from a development inter
Actions taken or work performed through vention; may also include changes resulting
which inputs, such as funds, technical assis- from the intervention which are relevant to
tance and other types of resources are mobi- the achievement of outcomes.
lized to produce specific outputs.
Outcome
The likely or achieved short-term and me
dium-term effects of an intervention’s out-
puts.
Impact
Positive and negative, primary and secondary
long-term effects produced by a development
intervention, directly or indirectly, intended
or unintended.
Source: OECD/DAC
Step 1: Define the project objectives
First of all, it must be clear what effect the project should have. There should also be analysis, together
with the target group, of their problems, the causes and their needs. The guidelines show what methodo-
logies exist, how to define outcome and impact objectives, and the areas that require particular attention.
The problem tree is
High child used to identify
mortality in problems and their
region x, y, z causes.
Children Children contract
contract malaria diarrhoea
Lack of effective Children are Mothers use Medical
medicine inadequately unclean drinking treatment is bad
protected from water
infection
Research Vaccine is Mothers know Clean water is Clinics are Lack of
into effective too expensive too little about too far away too far away medicine
medicine the links
halted
8 Outcome and impact assessment in international development
The project objective describes the desired direct, short – and medium – term effects that the project should
have on the target group. The project goal describes the long-term effects that the project is meant to set in
motion – or help to set in motion. It establishes the link between the project and the overarching goals, both
inside and outside the organisation. The guidelines show how to ensure that objectives are defined proper-
ly and aligned with each other, as this then makes it possible to translate them into measurable indicators.
Variant 1 ( e.g. EU )
Objective Indicator
Impact level Less children in x die from Child mortality in x < 5% ( by 2015 )
diarrhoea
Outcome level Improved access Walk to next well < 15 min for 90% of households in x ( by 2015 )
to drinking water
Output level Build wells 100 wells operational
The variant illustrated
above shows how objecti-
ves and indicators are de-
fined and used. The gui-
delines contain further
variants that are used in
practice, but these should
not be mixed up.
Short presentation of the Zewo guidelines for projects and programmes 9
Step 2: Develop a results model
If one is to measure and verify the effects of a project, it must first be clear how the effects are to be achie-
ved. The guidelines show how to find solutions, choose a strategy of intervention and develop a results model.
Child mortality
in regions
x, y and z falls
Less children Less children
contract malaria contract
diarrhoea
Effective Children are More mothers Better medical
medicine is better protected use clean care for children
available from infection drinking water
Research is Free vaccines Mothers more People have More children More children
re-started for children aware of better access to can be treated receive effective
the links clean drinking in time medicine
water
Health courses Build wells Install mobile Hand out The objective
chosen strategy for young health clinics medicine tree is used to
desired side effects mothers
develop different
no intervention solutions.
10 Outcome and impact assessment in international development
The Logic Model is a simple and commonly used way of displaying a linear chain of
cause and effect and of illustrating how a project will work. The guidelines show what it is suitable for and
what alternatives there are.
Example of a health course
Input Activities Output Outcome Impact
2 million CHF Provide Mothers More mothers Mothers
for education courses have attended understand increasingly
courses the links use clean
drinking water
Example of a simple
Logic Model.
Short presentation of the Zewo guidelines for projects and programmes 11
Step 3: Plan the outcome and impact
assessment
Even before the project is implemented, plans must be laid for how its effects are to be measured and
assessed. The outcome and impact objectives must be converted into indicators, and the methods used to
collect and analyse the necessary data must be planned. The guidelines demonstrate how to define indi-
cators, which qualitative and quantitative methods exist, what kind of comparison is suitable for what, and
what aspects require particular attention when assessing outcomes and impact.
In a before-and-after
comparison with a con-
trol group, the develop-
ment of the target
group is set against the
development of a con-
trol group that has bene-
fited from none of the
Target group project outputs. This
excludes external in-
Control group fluences and makes it
possible to observe what
would have taken place
without the project.
Start Middle End
However, this approach
involves a great deal of
work and is methodo-
logically challenging. The
guidelines also give
alternative approaches.
12 Outcome and impact assessment in international development
There are warnings
N. B. about pitfalls, stumbling
It is frequently the case that, in practice, only the final situation of a project is described. blocks and known
Yet, for an impact assessment, a simple description of the target group with no link to the sources of error, and
the essential points are
objectives, the initial situation or a control group is not sufficient.
highlighted.
IMPORTANT
A good indicator ought to be SMART:
• Specific: the indicator must be unambiguous and clear.
• Measurable: the indicator must be measurable and the costs for measurements
appropriate.
• Achievable: the target value given by the indicator must be achievable.
• Relevant: the information provided by the indicator should be relevant for
the project manager.
• Time-bound: the indicator must show when the objective ought to be achieved.
Source : European Commission, PCM Guidelines
Short presentation of the Zewo guidelines for projects and programmes 13
Step 4: Collect data
Ideally, the data needed for the outcome and impact assessment should be collected during project imple-
mentation. The guidelines show how to integrate outcome indicators into the project monitoring system,
what to pay attention to during data recording, and how to detect and correct deviations at an early stage.
Child health programme
Strategy of intervention Indicator Source 2009 2010 2011 2012
Impact Contributes to improved child Child mortality in Regions x, y National x:10% x:5%
health: child mortality falls and z falls from 10% to 2% statistics y:10% y:7%
z:10% z:12%
Outcome Mothers know about links Participants in the courses Video, analysis Good Satis-
can use the information by project factory
learned in a role-playing managers
game.
Ill children can be 95% of cases of children Case studies 89% 80%
successfully treated treated for diarrhoea are
successful.
Improved access to clean Walking time to nearest well Observation 50% 60%
drinking water < 15 minutes for 80% of
households
Output Courses 100 courses provided Project report 23 42
Cases treated 1,000 cases treated per year Treatment 955 1112
statistics
Wells 50 new wells in the region Project report 12
The logframe is a
Activites Provide courses
standardised table
Introduce mobile that summarises
health clinics
how the project is in-
Build wells tended to work as well
as its monitoring and
evaluation system.
14 Outcome and impact assessment in international development
Interim analysis of health course
Objective Indicator Target Perfor- Difference Comments Operative Consequences
mance measures at outcome level
O1 Use in role play Min. good Good – Analysis easy None
P1 N° of courses 1 (Pilot) 1 (Pilot) – None
P2 N° of 35 45 + 30% High demand Larger course Target group
participants groups larger than
assumed
Cost 10 000 15 000 + 5 000 Higher Check savings Implementati-
attendance, per course on might be less
higher costs day, apply efficient
for increased
budget
Deadline May 09 June 09 1 month Finding Need to train Outcome
late speaker additional objectives will
harder than speakers be achieved less
expected quickly than
planned due to
need to train
speakers
Legend for colour code
Result Measure
Satisfactory Plan reinforcement and ensure continuous success
Take steps to correct This example
Critical
Keep under observation shows what
Still uncertain
None needed an interim
As planned
analysis might
look like.
Short presentation of the Zewo guidelines for projects and programmes 15
Step 5: Evaluate the effects
The guidelines help project managers to consolidate and interpret the qualitative and quantitative data. It
shows when effects can be attributed to a specific project as well as how to aggregate the effects of dif-
ferent projects.
How it is done
Activities Project managers or external experts make comparisons and find out the
project’s effect on the target group using the available data. This task should
be carried out according to standard evaluation practice. The findings are
generally recorded in writing.
Questions Coming up with answers to the following question forms the fifth step in an
impact assessment:
• Is all the necessary data available in a suitable format?
• What was the effect or change on the target group?
• What would have changed for the target group without the project?
• What are the reasons for any deviation from the project objectives?
• Which assumptions and hypotheses have proved true, and which
were false?
• What foreseen and unforeseen side effects were there?
• Is there a plausible case to be made that the project has contributed
to the overarching goals?
• Which effects can be clearly attributed to the project?
• Which recommendations are needed?
Results A report or a presentation has been made about the effects of the project
or programme.
The chapter summary gives
an overview for each step.
16 Outcome and impact assessment in international development
Templates make work
easier for the users.
Template
for how to structure an impact assessment report
I Summary
II Basic principles
1. Rationale, purpose and objectives
2. Scope of the impact assessment
3. Questions for impact assessment
3.1 Question a
3.2 Question b
4. Context of the impact assessment
5. Team
III Approach
1. Discussion of methodology, sources of information
and data quality
2. Inclusion of relevant stakeholders
The outcome and impact assessment findings
IV Findings
1. Question a should always be reported in a suitable form,
1.1 Observations regardless of whether the findings are expec
1.2 Appraisal and conclusions ted or unexpected, negative or positive. This
2. Question b
2.1 Observations is often via a written report. The guidelines
2.2 Appraisal and conclusions show how these are generally structured and
V Overall conclusions and recommendations what other means of communication one might
envisage.
Short presentation of the Zewo guidelines for projects and programmes 17
Step 6: Use the findings
The assessment’s findings can be used to build up organisational knowledge and to learn lessons for the
future, to steer the organisation’s activities towards results or to inform donors, partner organisations and
the target groups about the projects’ and programmes’ effects.
Learning
Steering
Legitimising
18 Outcome and impact assessment in international development
The guidelines show what is required if the learning process is to succeed, how to use the findings as a basis
for decision-making, and how to include them in the organisation’s yearly performance report according to
Swiss GAAP FER. There is a clear summary of the principles of good outcome and impact assessment, which
also shows how much impact assessment should cost.
A template and practical
examples give suggestions
as to how to include
the results of the impact
assessment in the
performance report.
( Available in German
and French )
Short presentation of the Zewo guidelines for projects and programmes 19
Resources and links
The guidelines give clear summaries of the main approaches and methodologies for impact assessment.
There is also a practical and well-structured list of links and downloads on the subject of outcome and
impact assessment.
Approaches and methodologies
■■ Logical Framework Approach – is the most widespread model for results-based project planning in
development circles.
■■ Outcome Mapping – is an alternative approach to developing a system to record the (qualitative) effects
of projects and programmes.
■■ Theory of Change – is an approach for results-based project planning and is based on a somewhat more
open results model than the Logical Framework Approach.
■■ Most Significant Change – is a very specific, qualitative and participatory technique for recording the
effects of projects and programmes. It is based on the systematic analysis of individual experiences and
thus dispenses entirely with indicators and figures.
■■ MAPP ( Method for Impact Assessment of Programmes and Projects ) – is a specific, participatory method
for recording the effects of projects and programmes. It is based on group discussions during which effects
and developments are analysed retrospectively following a set programme.
Links
■■ Overviews of different methodologies
■■ Handbooks on monitoring and evaluation
■■ Toolkits with ready-to-use tools for specific cases
■■ Field reports
20 Outcome and impact assessment in international development
Good Practice
There is no single impact assessment system. Every organisation must develop a plan that is adapted to its
own circumstances. In order for this to succeed the guidelines set out a few basic principles for ensuring
assessment quality and for publishing and communicating results.
The Zewo Foundation would like to promote widespread adoption of systematic impact assessment in
practice and encourage aid agencies to develop and implement impact assessment systems that are tailored
to their needs as part of this good practice.
Short presentation of the Zewo guidelines for projects and programmes 21
Online guidelines
You can find the guidelines on-
line at www.zewo.ch/impact/en,
where they are also available as a
PDF file.
22 Outcome and impact assessment in international development
Supporting institutions
The Swiss Agency for Development and Cooperation ( SDC ) is the inter-
national development body within the Federal Department for Foreign
Affairs ( EDA ). It played an important part in developing and funding
these guidelines.
The Swiss Evaluation Society ( SEVAL ) fosters the exchange of infor-
mation and experience in the field of evaluation. It actively engages in
improving the quality of evaluation and its diffusion. SEVAL supports
the launch of these guidelines as a communications partner.
The ETH Zurich’s Centre for Development and Cooperation ( NADEL )
trains young and experienced professionals working in development co-
operation. In its courses NADEL teaches, among other things, methods
of results-based planning and impact analysis, as well as the Zewo gui-
delines for outcome and impact assessment.
We would like to thank the members of the working group – Bernard Du Pasquier ( HEKS ), Diether Grünenfelder
( EcoSolidar ), Maya Natarajan ( IAMANEH Switzerland ), Constanze Bunzemeier ( Enfants du Monde ),
Peter Schmidt ( Helvetas ), Gerhard Siegfried ( SDC ) and Christian Varga ( Caritas Switzerland), as well as
Oliver Bieri and Stefan Rieder from Interface Policy studies Research Consulting.
Short presentation of the Zewo guidelines for projects and programmes 23
Zewo Foundation
Lägernstrasse 27
8037 Zurich, Switzerland
Telephone +41 (0)44 366 99 55
Fax +41 (0)44 366 99 50
[email protected]www.zewo.ch
