COMPLEX COLLECTION AND

ANALYSIS METHODOLOGIES
Many different data collection tools and methods can be used for monitoring and evaluation. Some are
simple tools, which generate data that can be analysed or used in many different ways. Others are more
complex methodologies that include guidelines and processes for data selection, analysis and use, as well
as data collection.

Many different data collection tools and methods can be such as MSC and outcome harvesting, can be applied
used for monitoring and evaluation (M&E). Some are midway through a project or programme even if they have
simple tools, which generate data that can be analysed or not been planned from the start. Others, such as
used in many different ways. Basic tools include interviews, Randomised Control Trials (RCTs), need to be planned from
focus group discussions, observation, photography, video, the very start of a project or programme.
case studies, stories of change, surveys and questionnaires.
These tools are covered in a separate M&E Universe paper. Some of the methodologies are deliberately designed to
encourage the active participation of different stakeholders
There is also a wide (and growing) range of more complex within M&E. For example, MSC, PLA and outcome
methodologies. These typically include guidelines and harvesting are all participatory methodologies. On the
processes for data selection, analysis and use, as well as other hand, some of the methodologies are designed to be
data collection. Sometimes these methodologies are approached in a more top-down way, and require more
focused on gathering and analysing new data, and specialist knowledge to design and implement.
sometimes they are focused on analysing existing data, Methodologies in this category include RCTs, quasi-
perhaps supplementing it with new data if required. experimental approaches and Qualitative Comparative
Analysis (QCA).
There is no standard, accepted terminology for these
methodologies within M&E. Some call them methods or A list of the complex methodologies covered in the M&E
methodologies; others call them tools, techniques or Universe is described below.
approaches. For the purposes of this paper a complex
methodology is characterised by two things:  Outcome harvesting is an M&E methodology used to
identify, describe, verify and analyse outcomes. It
• multiple basic tools (such as interviews, involves collecting and recording multiple changes (the
observation and surveys) are used to collect data outcomes), grouping the changes into different areas or
(or data is used that has already been collected); domains, and then working backwards to assess
and whether and/or how an organisation, programme or
• there are prescribed guidelines or processes for project contributed to those changes. Outcome
analysing data. harvesting was partly inspired by Outcome Mapping,
and the two methods are often seen as complementary.
Some of the complex methodologies described in this A key difference is that Outcome Mapping is designed
paper were originally designed to be used during project or to be used during planning as well as during (or after)
programme design. For example, Organisational implementation of projects and programmes. On the
Assessment (OA) tools, Participatory Learning and Action other hand, Outcome harvesting can be used where
(PLA) and Appreciative Inquiry (AI) were all used to help there is no plan, or where a plan has become obsolete.
design and plan projects and programmes before they were
 The Most Significant Change (MSC) technique is a form
ever used as M&E tools. By contrast, methods such as Most
of participatory M&E that involves the collection,
Significant Change (MSC) and outcome harvesting were
selection and verification of stories of change,
designed specifically for M&E purposes, and can only
developed by programme or project staff and/or
normally be used once enough time has elapsed for
beneficiaries. It seeks to engage participants in a
changes to occur within a project or programme.
process of discussing, analysing and recording change.
The complex methodologies described in this paper are In a hierarchical organisation, MSC works by generating
fundamentally about assessing change at outcome and stories at different levels and passing the most
impact level. They may involve some analysis of activities significant stories up to the next level for further
and outputs, but they are primarily designed to assess the analysis. This thereby produces a set of most significant
difference that a project, programme or organisation stories of change at successive levels. MSC is most
makes in the medium- to long-term. Any of the methods useful when it is not possible to predict in any detail, or
can be used as part of ongoing M&E or during reviews, with any certainty, what the outcomes of a project or
evaluations and impact assessments. Some methodologies, programme will be at the start.

© INTRAC 2017
 Qualitative Comparative Analysis (QCA) is a is therefore seen as an alternative to more traditional
methodology that allows for the analysis of multiple forms of inquiry, which often focus on problems,
cases in complex situations, and helps explain why challenges and difficulties.
change happens in some cases but not others. QCA is
 Quasi-experimental approaches: Experimental
one of the few methodologies that uses both
approaches aim to measure changes within a
quantitative and qualitative analysis. It is designed for
population that receives products and/or services by
use with an intermediate number of cases, typically
comparing their situation with a comparison group that
between 10 and 50. It can be used in situations where
does not. In a full experimental approach, individuals,
there are too few cases for conventional statistical
households or organisations are randomly assigned to
techniques. It was originally used as a research
separate groups – some groups receiving the
methodology, but lately has increasingly been applied
intervention and others not. Quasi-experimental
within M&E. Sometimes QCA involves the collection of
approaches also use comparison groups, but the groups
new data. At other times QCA can be applied to data
are not allocated randomly. Quasi-experimental
that has been collected previously.
approaches are a form of quantitative inquiry, and aim
 Process tracing is a qualitative analysis methodology. Its to generate statistically significant findings.
main purpose is to establish whether, and how, a
 Randomised Control Trials (RCTs) are the most
possible cause or causes influenced a specific change. It
common type of full experimental approach used in
does this by applying a set of formal tests to examine
social development. Like quasi-experimental
the strength of evidence linking the potential causes to
approaches they result in quantitative assessment of
a change or set of changes. A key feature of process
change. They are deliberately designed to address
tracing is the development and testing of alternative
selection bias, which occurs when a group receiving an
ideas about how a change might have come about. It
intervention is systematically different in some way
was originally used as a research method that
from a comparison group. RCTs avoid this bias by
attempted to provide theoretical explanations of
randomly deciding who receives products and services
historical events, but nowadays it is increasingly being
and who does not (the control group). RCTs were
used within M&E. CSOs tend to use process tracing in
mostly used within research projects until quite
areas of work such as policy influencing, where change
recently, and were rarely used for M&E or impact
may be the result of many different influences.
assessment. They suddenly became popular in the early
 Contribution analysis is similar to process tracing in 21st century, and, controversially, are sometimes seen
many ways. Like process tracing it is a methodology that as the ‘gold standard’ for impact assessment.
attempts to identify the contribution a development
 Participatory Learning and Action (PLA) is a type of
intervention has made to a change or set of changes.
qualitative research, which can be used to gain an in-
The ultimate aim of contribution analysis is to produce a
depth understanding of a community or situation. It is
credible, evidence-based narrative of contribution that
widely used in work involving local communities. PLA is
a reasonable person would be likely to agree with,
a participatory methodology, and should always be
rather than to produce conclusive proof. Contribution
conducted with the full and active participation of
analysis is normally only used in a project or programme
community members. The main purpose of PLA is to
after enough time has passed for change to occur.
support people within communities to analyse their
 Tracer studies involve collecting and analysing data own situation and to ensure that any learning is then
repeatedly over a period of time. They are sometimes translated into action. In its true form, PLA is a
called tracker studies or longitudinal studies. They are philosophy and a way of thinking that encourages
normally designed to track changes on individuals, reversals in power relations between communities and
following a development intervention such as outsiders. It comprises a range of participatory exercises
vocational training. But tracer studies may also focus on such as focus group discussions, interviews, mapping,
other units of analysis, such as households, ranking, timelines, calendars and diagrams, which can
communities, organisations, projects and policies. be used to work, plan and reflect with and alongside
Tracer studies may involve qualitative or quantitative communities.
inquiry, or a mixture of both. They are most useful
 Organisational assessment (OA) tools, also known as
when change from an intervention is intended to be
organisational capacity assessment tools (OCATs), are
long-term and significant. Tracer studies help show not
designed to assess organisational capacity and plan
just whether a change occurs, but when it occurs as
capacity development initiatives. Sometimes they are
well.
used for M&E purposes as well. They are the only M&E
 Appreciative Inquiry is a participatory form of inquiry tool in widespread use designed specifically with
that influences how information is collected, analysed capacity development in mind. OA tools can be used for
and used. It was originally designed to support work M&E in two ways. Firstly, an organisational assessment
involved with organisational change, but can be used to can lead to the development of a capacity development
help design, plan, monitor and evaluate any kind of action plan with associated objectives and indicators
development intervention. Appreciative Inquiry that can be monitored over time. Secondly,
deliberately focuses attention on what is working well organisational assessments can be repeated at discrete
and how things could be made even better, rather than intervals to show whether or how capacity has changed
focusing on what is going wrong or needs to be fixed. It within an organisation.
© INTRAC 2017
 Social Network Analysis (SNA) comprises a range of mobile data collection, remote sensing and real-time,
different tools and methods designed to help map and simple reporting.
analyse social networks. It is mainly aimed at identifying
 Two papers in the planning section of the M&E Universe
and analysing the relationships within and between
deal with the logical framework and Outcome Mapping.
different components of a social network. SNA can be
Both of these methodologies are designed to be used
applied to networks at any level from local to
when a project or programme is being planned, but
international. SNA is commonly used in areas such as
both have implications for data collection and analysis
policy influencing and mobilisation, where significant
during implementation as well. The logical framework
amounts of work are carried out through partnerships
tool sets out what needs to be monitored and
and coalitions. SNA has also been used extensively to
evaluated, but does not prescribe any methods. On the
map and analyse knowledge networks and communities
other hand, Outcome Mapping comes with a set of
of practice.
procedures and methods for collecting and recording
information over the course of a programme.
Other methods for collecting and  A paper in the analysis section of the M&E Universe
analysing data deals with cost-benefit analysis, which is usually used
for design and planning but is sometimes used for M&E
There are a number of other papers in the M&E Universe as well.
dealing with complex methodologies for collecting and
analysing data. These are as follows. Many other tools and methodologies are available but not
covered in the M&E Universe. Many are variants on a
 A paper in this section briefly covers some of the newer theme, and many involve merging two or more existing
M&E tools and methods, which have been driven by tools and methodologies. M&E practitioners that are able
advances in Information and Communications to understand the complex methodologies described in this
Technology (ICT). These include beneficiary feedback section of the M&E Universe should also be able to
mechanisms (BFMs), crowdsourcing, data exhaust, data understand any other conceivable method.
visualisation, intelligent infrastructure, micro-narratives,

Further reading and resources

Links to further papers on all the tools and methodologies mentioned in this paper can be found below.

Outcome harvesting Most significant change

Qualitative comparative analysis Process tracing

Contribution analysis Tracer studies

Appreciative inquiry Quasi-experimental approaches

Randomised control trials Participatory learning and action

Organisational assessment tools Social network analysis

Outcome Mapping The logical framework

ICT in monitoring and evaluation Cost-benefit analysis

© INTRAC 2017
 Author(s): INTRAC is a not-for-profit organisation that builds the skills and knowledge of civil society
INTRAC organisations to be more effective in addressing poverty and inequality. Since 1992 INTRAC has
provided specialist support in monitoring and evaluation, working with people to develop their own
M&E approaches and tools, based on their needs. We encourage appropriate and practical M&E,
based on understanding what works in different contexts.

INTRAC Training M&E Universe

M&E Training & Consultancy M&E Universe
We support skills development and learning on a range of For more papers in
INTRAC’s team ofhigh
themes through M&E specialists
quality offer consultancy
and engaging and
face-to-face, For
themore
M&Epapers in
Universe
training in all
online and aspects oftraining
tailor-made M&E, from
andcore skills development
coaching. the series
M&E Universe
click the
through to the design of complex M&E systems. series
homeclick the
button
Email: [email protected] Tel: +44 (0)1865 201851 home button
Email: [email protected] Tel: +44 (0)1865 201851

© INTRAC 2017