Theme: Building health information systems

Role of health surveys in

national health information
systems: best-use scenarios

Theme 1: Building health information systems

health system information for decision-makers

Health Information Systems Knowledge Hub

Institute of Health Metrics and Evaluation

Working Paper SeriesNumber 6November 2009WORKING PAPER

Strengthening health systems

in Asia and the Pacific through
better evidence and practice

For the PDF version of

this paper and other
related documents, visit
www.uq.edu.au/hishub

About this series

The Health Information Systems Knowledge Hubs
Working Paper Series is the principal means to
disseminate the knowledge products developed by the
hub as easily accessible resources that collectively form a
lasting repository of the research findings and knowledge
generated by the hubs activities. Working papers are
intended to stimulate debate and promote the adoption
of best practice for health information systems in the
region. The series focuses on a range of knowledge gaps,
including new tools, methods and approaches, and raises
and debates fundamental issues around the orientation,
purpose and functioning of health information systems.
Generally, working papers contain more detailed
information than a journal article, are written in lessacademic language, and are intended to inform health
system dialogue in and between countries and a range of
development partners.
Many working papers have accompanying products such
as summaries, key points and action guides. The full range
of documents, as well as other resources and tools, is
available on the Health Information Systems Knowledge
Hub website at www.uq.edu.au/hishub/publicationtools.
The opinions or conclusions expressed in the Working
Paper Series are those of the authors and do not
necessarily reflect the views of institutions or governments.

The University of Queensland 2009

Published by the Health Information Systems
Knowledge Hub,
School of Population Health
The University of Queensland
Public Health Building,
HerstonRd, Herston Qld 4006, Australia
Please contact us for additional copies of
this publication, or send us feedback:
Email: [email protected]
Tel: +61 7 3346 4732
Fax: +61 7 3365 5442
www.uq.edu.au/hishub
Design by Biotext, Canberra, Australia

Acknowledgments
Many individuals and institutions with expert
knowledge of health information systems in the
AsiaPacific region have generously assisted the hub
in the preparation of this document. The Institute
for Health Metrics and Evaluation (University of
Washington) and the Health Information Systems
Knowledge Hub (School of Population Health, The
University of Queensland) are grateful to Rafael
Lozano, Ali Mokdad, Michael MacIntyre, Kelsey
Moore, Tim Adair, Maxine Whittaker and Alan Lopez
for their contributions.

Author details
Health Information Systems Knowledge Hub,
School of Population Health,
The University of Queensland
Institute of Health Metrics and Evaluation,
University of Washington

Contents
Acronyms and abbreviations 2
Introduction 3
Essential health system metrics 5
Key considerations for country survey implementation18
Survey design and data collection 20
Conclusion 23
References 24

Health Information Systems Knowledge Hub

Working Paper SeriesNumber 6November 2009

Acronyms and abbreviations

Behavioural Risk Factor Surveillance System

DALY

disability-adjusted life year

DHS

demographic and health surveys

GAVI

Global Alliance for Vaccines and Immunisation

HIV

human immunodeficiency virus

ICD-10

International Statistical Classification of Diseases and Related Health Problems, 10th revision

IHME

Institute for Health Metrics and Evaluation

MDG

Millennium Development Goal

MICS

multiple indicator cluster surveys

NHANES

National Health and Nutrition Examination Survey

SF-36

Medical Outcome Study Short Form Health Survey 36

UN

United Nations

UNICEF

United Nations Childrens Fund

US

United States

USAID

United States Agency for International Development

WHO

World Health Organization

YLD

year lived with disability

Health Information Systems Knowledge Hub

BRFSS

Working Paper SeriesNumber 6November 2009

Introduction
Health surveys are a fundamental component of a wellfunctioning and complete health information system.
Although they are not ideal for capturing all health
information necessary to manage the health system,
in developing countries particularly they have a role to
complement or supplement other health data sources.
Governments can use data gathered from surveys to
understand key health issues within their own country
and to act in consequence. To further advance their
efforts at accountability, many developing countries
are attempting to move towards an evidence-informed
policy-making model. As a result, methods for assessing
health outcomes need to be time-efficient to conduct,
flexible to specific targeting, and provide accurate results.
Furthermore, the need to assess outcomes before
implementing health reforms is becoming increasingly
important as international agencies and sponsors
prioritise monitoring and evaluation as a proportion of
their investments (Evans et al 2005). Health surveys are
an important way to identify, capture, and manage a
growing evidence base for health.

Health information systems consist of many different

components. At core, a health information system is
the structure that supports the production of health
statistics and health information. This paper summarises
the critical information requirements of any health
information system, with a focus on one element
of these systems: surveys. It examines the ability of
countries with limited financial, infrastructure and
human capacity to calculate essential health metrics
by supplementing their current health information
collections with data collected from surveys.
Within the health information system, surveys can play
multiple roles. They can provide information on subsets
of populations where full information is not available.
Alternatively, they can be used to supplement data
where the only information comes from predicted figures
based on models. In either case, surveys help to provide
better information and drive better decision-making.
Every health system has a key set of essential health
metrics that should be measured regularly to inform
decision-makers how the health system is performing
and how it can be improved. In an ideal setting, these
essential health metrics are gathered in a coherent and
coordinated way, with precision, through the health
information system. However, resource constraints and
other practical realities dictate that not all countries
can achieve this ideal. This working paper outlines
these essential health metrics, the critical purpose
they serve and how they might be measuredwith a
particular focus on surveys. We summarise some of the
core strengths and weaknesses of each mechanism for
information gathering. This working paper is intended

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Recently, leaders of the World Health Organization

(WHO), the Global Fund to Fight AIDS, Tuberculosis
and Malaria, the Global Alliance for Vaccines and
Immunisation (GAVI), the United Nations (UN)
including the UN Population Division and the United
Nations Childrens Fund (UNICEF)the World Bank
and the Gates Foundation concluded that substantial
increases in international attention to health have been
accompanied by demands for statistics that accurately
track health progress and performance, evaluate the
impact of health programs and policies, and increase
accountability at country and global levels (Chan et al
2010, p 1). They recognised that surveys, particularly
household health surveys, play a valuable role in low and
middle-income countries. Surveys are the main source
of data for monitoring progress toward several of the
UNs Millennium Development Goals (MDGs), including
providing a more complete picture of health outcomes,
risk factors, service coverage and equity. Recognising this
important role, efforts have been made to harmonise
international surveys, such as demographic and health
surveys (DHS), multiple indicator cluster surveys (MICS),
and other epidemiological surveys, as well as to increase
their frequency, to improve the availability of comparable
data across countries and time. These efforts have
helped, but there remain gaps and areas for substantial
improvement.

Problems that have arisen as a result of information

failures, and the actions that resulted in these failures,
are well documented (Friedman and Wyatt 1997, Heeks
2002, Littlejohns et al 2003, Gauld and Goldfish 2006)
but there is little information on solutions and successes
(Kuhn and Giuse 2001, Van der Meijden et al 2003).
These failures represent the experiences of those who
work in the field, or the supply side of health information
systems. This working paper is intended for those on
the demand side of health information systems, which
includes decision-makers and funders. It is intended to
provide guidance regarding the usefulness of surveys in
specific situations, to improve the quality of policies and
investments in health information.

Health Information Systems Knowledge Hub

as a high-level guide for decision-makers so they

understand their options and can make sense of them.
We then provide some guidance on how decision-makers
can engage other important stakeholders to understand
the importance of health information development and
some of the key aspects that they should consider in
implementing health information systems reform.

Working Paper SeriesNumber 6November 2009

Essential health system metrics

While there is a plethora of health indicators that
countries might wish to collect, the critical question for
health managers is, what is the minimum information
that I need to collect to manage my health system, at
different levels, and what is the most cost-effective
means to collect that information? Managers also need
to be confident in the reliability of data, be able to use
recent data on the health situation, and make choices
that promote the sustainability of the health information
systems upon which they rely.

components over time. Furthermore, there will be

trade-offs between the completeness of information in
each of these areas. Those trade-offs must be weighed
contextually, with consideration of gathering information
across all areas.
Finally, surveys are not always the answer. They can
certainly provide useful information, but they are by no
means the only source of data, as is explained below.

Table 1 illustrates the common sources of data (columns),

and lists a series of essential health metrics that should
be routinely monitored as part of a comprehensive
health information system (rows). Not all emerging
systems will have the resources to track all the metrics
listed, and not all the data sources can provide equally
useful information. For each of the substantive health
metrics components, we have indicated the optimal
source of information and any possible sources. Optimal
data sources are those that will provide the most
complete and relevant information to meet the needs
of that particular metric. Possible sources are those that
can provide partial or slightly less verifiable information
for a particular metric, but that is useful, nonetheless.
These are general guides, and in the following section
we discuss in greater detail some of the dimensions and
key aspects of each metric to broadly describe the merits
of each, and the approaches that decision-makers might
take in using them.

what exists (and therefore, what can be built on)

what they can afford to establish
what sequence of implementation is best
most importantly, how they can knit together the
multiple sources available to them for the best
possible overall picture.
Decisions of this nature may well vary between contexts
and systems.
Second, collecting high-quality information on each
of the essential metrics should be the ultimate aim;
however, in some contexts it is prudent to stage

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Three important considerations should be borne in

mind to understand the merits and limitations of the
information presented here. First, for any individual
health information system, the local decision-makers and
those who know the context best must make decisions
about:

Working Paper SeriesNumber 6November 2009

Summary health metricdraws from multiple metrics above

Possible

Possible

Possible

Possible

Census

Optimal

Optimal

Optimal

Optimal

Optimal
Possible

Possible

Possible

Possible

Possible

Possible

Possible

Epidemiological
Health facility
observational studies assessments

Optimal

National health
accountsb, budgets
and expenditure
reports

a These metrics require a combination of data sources to provide the most complete data. It is not possible to generate optimal information without pooling the data from these various data sources.
b National Health Accounts
Blank cells indicate metrics for which no data sources are recommended.

Optimal

Health system
performance
assessment

Possible

Summary health metricdraws from multiple metrics above

Possible

Burden of disease/
comparative risk
assessment

Optimal

Possible

Possible

Possible

Possible

Possible

Possible

Hospital
discharge

Health servicesa

Optimal

Optimal

Optimal

Possible

Health service
registry

Optimal

Possible

Optimal

Possible

Health examination
surveys

Effective coveragea Optimal

Cost-effectiveness

Health financing

Human resource
inputs

Functional health
status

Morbidity

Possible

Optimal

Cause of death

Risk factors

Optimal

Mortality

Possible

Household
Vital registration interview surveys

Primary microdata source

Essential health system metrics and potential data sources

Metrics

Table 1

Health Information Systems Knowledge Hub

The sources of data listed in Table 1 cover a range of

health indicators that any health information system
could potentially generate. The types of indicators
produced should be clearly and closely aligned with
the needs of decision-makers. There are also important
health characteristics that all health information systems
should measure at the population level, including those
related to health outcomes, mortality, causes of death,
functional health status, risk factor exposure and the
burden of disease. Other important metrics are related
to health services provision, such as effective coverage,
and resources inputs that are measured by human
resources and financing. Finally, for system-wide and
intervention-specific evaluations, it is important to
include health metrics related to cost-effectiveness or
health system performance assessment.
It is important to consider the potential uses of each
of these essential health metrics. Decision-makers will
often be interested in the levels of such indicators, as
well as the trends over time. They will also be interested
in any inequalities that exist within the population; for
example, disparities in mortality by sex, or differences in
the effective coverage of priority interventions by income
or population subgroup. Such data would often need
to come from multiple sources. Hence, data availability
should be a key consideration when developing the
scope of each health metric.

Ideally, any health information system will provide

reliable information on each of the essential health
metrics described below. Practically, this will not
always be possible, but we hope that decision-makers
will take as their goal the need to assemble the most
comprehensive picture of the operation, and outcome,
of their health system.

Mortality
The basic mortality information required is the number
and distribution of deaths in a population by age and
sex. This information can be used to compute basic

A complete vital registration system is the single best

source of mortality data. The main source of vital
registration data is civil registration. Civil registration
data are collected as part of nationally mandated
programs that are carried out primarily for legal and
governmental purposes. Collection involves completing
birth and death certificates for each person, with basic
information such as the persons name, sex, and date
and place of birth or death. Most death certificates also
list the cause of death using International Statistical
Classification of Diseases (ICD-10) codes. One of the main
advantages of civil registration is that it is a continuous,
permanent, compulsory and universal recording of the
occurrence and characteristics of vital events pertaining
to the population. The events of interest for measuring
mortality are live births, deaths and foetal deaths (WHO
and HIS Knowledge Hub 2010).
Vital registration systems take considerable time and
money to bring to scale. A number of countries have
vital registration systems, but many such systems are
incompletethey do not cover the entire population
and are often non-representative. In areas where vital
registration is incomplete or missing altogether, and
there are no resources to rectify this situation in the
short term, several alternatives can provide useful
information on the level and age patterns of mortality, as
described below.
Surveys and censuses can be used to collect accurate
mortality data, especially in resource-poor settings.
Household surveys such as the DHS or MICS use a
nationally representative sample and measure a range of
health indicators that are comparable across countries.
Surveys are particularly useful for collecting under-five
mortality data, because they use birth histories reported
by mothers. The type of data available in surveys ranges
from detailed birth histories with information such as
date of birth and death, to questions of the number
of children ever born and surviving for each mother.
Different methods have been developed to convert
various types of data into under-five and infant mortality
rates (Preston et al 2001). However, given that these
methods rely on retrospective reporting by mothers, the
resultant mortality rates may be subject to recall bias.

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

In the following sections, we describe the key elements

for which decision-makers need information to manage
the health system. Accompanying each essential health
system metric is a description of the optimal data source
that should be used, as well as other sources that could
possibly provide data, but would present barriers to
collection or other challenges.

mortality measures, such as child and adult mortality

rates and life expectancy, which are major indicators
of population health.

Health Information Systems Knowledge Hub

Household surveys have also been useful for providing

data on adult mortality. Mortality questions are
commonly asked about household deaths in the past
12 months, and about parental, spousal and sibling
survival (Gakidou et al 2004). Maternal mortality rates
and ratios can also be measured using data on the
survival of sisters from such surveys. Household surveys
generally also provide data on a variety of topics such as
socioeconomic indicators and health indicators, which
allow mortality rates to be contrasted between different
population groups. They are also of great importance
for obtaining mortality data in emergencies and after
disasters. However, estimates of adult mortality rates
are again subject to recall bias from retrospectively
reported data, as well as potential measurement error
due to clustering of mortality in families. Furthermore,
household surveys are subject to sampling error because
they use a sample. Surveys are also implemented
infrequently, which can make time trends difficult to
establish, and the costs associated with more frequent
surveys can be prohibitive. If data are collected over
multiple rounds, careful consideration should be given
to which questions are asked repeatedly to provide
comparability of movement over time.

Population censuses are a primary source of information

about a countrys population. Because they collect data
either for the population or for a representative sample,
they are especially advantageous for providing mortality
data for specific geographic areas. However, because of
the long interval between censusesoften a decade or
morethey are not ideal to gather data on time trends,
since decision-makers generally require more up-to-date
information. Population censuses use similar questions
that are applied by surveys for measuring child, adult and
maternal mortality.
Mortality estimates are often best conducted using
multiple data sources, such as the assessments of
under-five mortality produced by UNICEF, WHO, the
UN Population Division and the World Bank. These
assessments include information from complete vital
registration systems, partial vital registration systems,
birth histories from the DHS or similar surveys, survey
or census questions on the number of children ever
born and surviving, epidemiological surveys and sample
registration systems. Over the past decade, WHO,
UNICEF, the UN Population Division and the World Bank
have not always agreed on past estimates, predictions for
a recent period or forecasts. Advances have been made
to put all data sources used for tracking child mortality

by these agencies in the public domain and to harmonise

the work of identifying past trends and generating
current estimates across agencies. Despite these efforts,
international agencies still do not always include all
available local data (Murray 2007).
Epidemiological observational studies are also a viable
option for calculating population mortality levels and
patterns, because they attempt to capture data on
the cause of death, mortality and frequently fertility.
These studies are not regularly conducted and are
generally focused more on specific diseases or specific
indicators. Therefore, decision-makers should not use
epidemiological observational studies solely to measure
mortality levelsalthough if they have already been
done for other purposes, they may provide useful data.

Cause of death
Similar to mortality data, a fundamental aim of any
health information system should be to provide
an understanding of the distribution of causes of
death across a population by age and sex. The key
consideration with cause-of-death statistics is to reliably
capture the underlying cause of death for each death in a
population in a way that is comparable. Such information
is a critical input into public health planning, program
implementation and evaluation. Understanding patterns
of causes of death can help decision-makers to identify
the causes that can be mitigated, if not prevented.
This can facilitate more targeted resource allocation to
improve the overall health of a population. Most highincome and many middle-income countries benefit from
a complete vital registration system where most deaths
have a certificate of cause of death completed by a
physician using ICD-10 codes.
Vital registration systems are the gold standard for
obtaining cause-of-death data, although their use has
some limitations, particularly in developing countries
where there is incomplete registration of death and
poor quality of cause-of-death recording. If no medical
certificate exists, verbal autopsy (interviewing relatives
to ascertain probable cause) is the next best option.
Verbal autopsies rely on the assumption that signs
and symptoms associated with major causes of death
can be reliably recalled by a relative or caretaker
of the deceased after a death. Sociodemographic
information about the deceased is also collected from
relatives or caretakers. Verbal autopsies can be used to

Working Paper SeriesNumber 6November 2009

augment an incomplete vital registration system, with

the cause of death assessed from the verbal autopsy
replacing, in some cases, more formal but less reliable
vital registration data. Verbal autopsy can also be
used to validate the information recorded by the vital
registration system. In this case, the cause assigned
on the death certificate is compared with the cause
derived from the verbal autopsy. Some countries have
mandated this approach for certain types of death. For
example, Brazil has decided to use verbal autopsy in
areas of the country with consistently poor data from
vital registration systems and where there is a high
proportion of ill-defined causes of death. The Ministry of
Health in Brazil trained data collectors in verbal autopsy
methods for these poorly counted areas and is currently
incorporating the data with vital statistics. However,
while verbal autopsies can be useful for substantiating
vital registration data, they are better at differentiating
between some types of deaths than others. Therefore,
such assessments may skew the actual cause-of-death
pattern across a population, but still be substantially
better than having no data at all.

Abbreviated (shortened) verbal autopsies, much like

general surveys, have also been used in refugee camp
settings and emergency situations. The findings from
abbreviated verbal autopsies are often used to trigger
interventions. For example, a recorded increase in
mortality from measles should trigger a vaccination

When vital registration systems are incomplete and

verbal autopsies are not an option, cause of death
can also be determined from hospital discharge data.
Hospital data collection systems exist in many countries,
and while more reliable information is available for
deaths certified in hospitals, deaths in hospitals are
not representative of deaths in the population. For
example, some causes of death are less likely to occur
in a hospital: in many cultures, people with long-term
chronic illnesses are frequently discharged when
treatment can no longer help them, and the person may
prefer to die at home. Also, those who die from certain
causes, such as road traffic accidents, may be less likely
to make it to the hospital before death occurs. Thus,
hospital discharge data are not representative of the
entire population, particularly in developing countries
where access to health care facilities is not equal across
the population (Mathers et al 2001). Socioeconomic
status and population access can thereby reduce the
probability of certain subpopulations dying in hospital.
Although deaths recorded in hospitals are not a random
subsample of all deaths in the community, they can still
be a valuable source of population health information.
Statistical methods exist to estimate population-level
cause-of-death patterns using regional vital registration
data to overcome the selection bias of hospital death
data (Murray et al 2007).
Household surveys in resource-poor settings are
generally not a feasible method for capturing complete
cause-of-death information across a population, because
they require several questions to elicit symptoms that
can be used to diagnose the underlying cause of death.
They are sometimes able to incorporate a modified
verbal autopsy to capture some broad cause-of-death

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Responses on the verbal autopsy questionnaire are

assessed using one of several methods. A commonly
used practice is for physicians to review the responses
to the questionnaire and assign a cause of death
accordingly. Increasingly, algorithmic and statistical
methods are being applied to the patterns of responses
from multiple verbal autopsies to attain aggregate
estimates of population-level cause-of-death patterns.
Some of these methods perform well and could be
used cost-effectively with better results than timeconsuming, expensive and sometimes idiosyncratic
physician reviews (Murray et al 2007). Verbal autopsies
can be incorporated into household surveys and used
in any instance where a death within a preset time
period (eg 12 months) has been identified. They can
also be used independent of household surveys but
as part of an ongoing surveillance system, as has been
done in a number of sites across Africa and Asia by the
International Network for the Demographic Evaluation
of Populations and their Health in Developing Countries
(Adjuik et al 2006).

campaign in a camp, while an increased mortality rate

from diarrhoea will focus responses on water and
sanitation. Combining both mortality and cause-ofdeath data into one survey is potentially very useful in
resource-poor settings. The Population Health Metrics
Research Consortium is a good example of a project that
is working to develop household surveys that incorporate
new and innovative ways of gathering accurate
information on mortality and causes of death (Population
Health Metrics Research Consortium Project 2010).
Verbal autopsies are currently conducted as part of the
regular health information system in Thailand, in India as
part of the Sample Registration System and in China with
the Disease Surveillance Points System (Gajalakshmi and
Peto 2004, Yang et al 2005).

information, such as questions that identify maternal

deaths using the sibling history method, which is
commonly used in demographic and health surveys in
countries without vital registration (Obermeyer et al
2010). Furthermore, unless they are conducted regularly,
surveys cannot provide time trends in causes of death.
Similarly, a census is not appropriate for collecting causeof-death information, because it would be prohibitively
expensive.
Occasionally, epidemiological surveys can be used
to provide cause-of-death information, but this is not
the optimal way to provide information for an entire
population. Epidemiological surveys tend to focus
on specific causes of death, with a risk of 'over
diagnosing' these diseases. Moreover, they are not
conducted regularly.

Health Information Systems Knowledge Hub

Surveys are by far the best means for assessing

behavioural risk factors, such as smoking, alcohol use and
sexual behaviours. Nutrition surveys are the only widely
used source for measuring dietary intake in a population,
and they are often also used to assess nutritional status
for maternal and child health. Commonly collected
indicators, such as weight for height or weight for age,
have been used to assess chronic or acute malnutrition
inchildren.

A number of risk factors affect health outcomes.

These risk factors span a range of categories, including
behavioural, environmental, genetic, metabolic,
nutritional, reproductive and occupational risks.
Countries can markedly improve the overall health of
their population by controlling, or at least reducing,
population exposure to these risk factors. Populationlevel exposure to these risk factors has been assessed
widely using surveys; reliable and valid trend data have
been produced as a result. Results from such surveys
have been used for planning intervention programs and
future health system needs. Health examination surveys
(in which a set of interview questions is paired with a
physical examination) are recommended for collecting
data on some risk factors. By gathering diagnostic
information, some standard of comparison for certain
risks (such as obesity) can be attained. This eliminates
the errors of self-reporting that accompany household
interviews, and enables researchers to gather biodata
that cannot be collected in the home.

The United States (US) Behavioural Risk Factor

Surveillance System (BRFSS) is a good example of a data
collection system for surveillance of key risk factors,
using a telephone survey. The BRFSS started in 1981 as a
point-in-time survey. From 1984, 15 US states participated
in an ongoing surveillance system with their interviews
spread across the whole year. Data were collected on
the six individual-level risk factors associated with the
leading causes of premature mortality among adults:
cigarette smoking, alcohol use, physical inactivity, diet,
hypertension and lack of seat belt use (Mokdad 2009).
The BRFSS has become the main source of information
for a wide range of risk factors in the US, having
expanded beyond its original set of indicators. It has now
become a model survey for a number of countries that
have each adapted the instrument for their own needs,
to good effect. However, even such a well-established
and widely emulated survey has important caveats. For
example, telephone-based surveys are associated with
self-reporting biases, which need to be controlled for
during data analysis. Additionally, access to telephones
among the population is essential for a telephone
household survey to work; cultural acceptance of
answering questions over the phone related to personal
health is also essential. In some settings, it may be better
to use door-to-door (in-person) surveys.

Health examination surveys are more expensive than

simpler interview surveys; therefore, the latter are
often used more widely. Simple interview surveys are
used to capture a broad range of risks. Encouragingly,
some countries such as Thailand, Mexico, Brazil, Chile
and Colombia have incorporated biomarkers into their
household interview protocols, such as dried-blood
spot collection. While not equivalent to health
examination surveys performed in a clinical setting,

Censuses have occasionally been used to collect

information about risk factors; for example, questions
about smoking and alcohol consumption are often
included in censuses. However, because collecting risk
information is not the primary purpose of a census, their
utility in this regard is limited. Additionally, questions
on censuses are usually expensive to run, and each
additional question increases the cost. Census data
can help understand population-level risks, because

Risk factors

10

innovative mechanisms for gathering biodata, such

as using dried blood spots, allow even more data to
be gathered within a household interview without a
great deal of additional cost.

Working Paper SeriesNumber 6November 2009

Functional health status

they can provide additional information on the social

determinants of health and the extent of population
inequalities, such as geographical or socioeconomic
disparities.

Morbidity
Morbidity measurements focus on estimating the
relative prevalence (new and old cases) of a particular
disease. Disease registries and clinical databases are
the most common source of morbidity data, but often
only cover a subset of the population. The breadth
and scope of these data sources can be expanded
dramatically through selectively implemented health
surveys to gather additional information on morbid
conditions. Routine household surveys that monitor
morbidity are a good building block for providing data
to monitor health programs in a country or community.
Surveys can also help us to understand the differences in
morbidity according to other population characteristics,
such as sociodemographic indicators that are not likely
to be reliably captured in administrative or clinical
records. Such data are valuable for the planning and
implementing of prevention programs.

Hospital discharge data can also be used to capture

morbidity, although the data are often biased. For some
conditions, only the more severe cases are admitted to
hospital, while for others, virtually none are. Moreover,
hospital discharge data are a biased source of morbidity
patterns, because people who live further from a hospital
are less likely to seek care, as are those without the
means to pay for services. Ideally, a hospital discharge
system will be part of an electronic medical records
system in place across all health facilities. This provides
the best information on morbidity, because patient
tracking is possible over the length of their interaction
with the health system, along with an exit examination
recorded in the discharge data.

Within high-income countries, especially in the context

of clinical trials, substantial work has been done to
measure functional health status. It is measured by
performance across a series of core health domains, such
as vision, hearing, cognition, mobility, pain, self-care,
affect and usual activities. In ideal settings, these tests
are administered as part of a larger health examination
survey, which then pairs the tests with questions about
the individual and their health in general. A popular,
though extremely expensive, version of a health
examination survey is the National Health and Nutrition
Examination Surveys (NHANES) administered by the
US National Center for Health Statistics to a nationally
representative sample of adults in the US. It involves a
full physical examination and a battery of physical tests
in conjunction with a set of interview questions. While
other countries have drawn from the gold standard set
of tests that are administered in NHANES, the overall
model is prohibitively expensive for most contexts. Even
in a highly developed country like the US, the sample
of individuals it reaches is very small because of cost
considerations.

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Self-reported conditions are often not validated against

gold standard clinical data, but can at least provide a
starting point for estimates of morbidity from certain
conditions at the population level. The data are likely
to be even more valuable if morbid conditions can
be verified through health examination surveys. Such
surveys also serve as tools to facilitate diagnosis,
because they usually submit the individual to a range of
diagnostic procedures.

Death rates and morbidity information do not tell the

whole story about population health. The functional
ability of those who have been sick or disabled through
disease or injury is also an important measure of
population health. Essentially, we want to know the
extent of disability that follows disease and injury
so that we can prioritise interventions to prevent
or reduce that harm. To that end, functional health
status gathers information on an individuals physical
and mental ability and how this can be compared. For
example, a person may experience chronic back pain
and this may detract from their overall health and
should therefore be measured by the health information
system. Chronic back pain may take away from their
ability to perform certain tasks. We consider this a loss
of health function, rather than premature death. For
many conditions, such as blindness, mental disorders or
musculoskeletal diseases, the main health effects are loss
of health function. Years lived with disability (YLDs) is a
commonly used metric summarising functional health
status (Murray and Lopez 2006). Approximately 37% of
disability-adjusted life years (DALYs) worldwide are due
to YLDsand are as high 56% in North America (WHO
2002).

11

Health Information Systems Knowledge Hub

Fortunately, there are a number of tests to measure

actual ability within key domains that can be easily
influenced with diagnostics that are simple to administer
and that do not require a clinical setting. For example,
simple tests like walking 100 metres to test mobility,
administering simple eye chart examinations, and
asking a person to lift an object of a certain weight can
be administered outside a clinical setting as part of a
household interview. Increasingly, low-cost technology
are being developed to monitor physical activity as well,
such as pedometers that can be easily incorporated into
cell phones. These offer exciting options for decisionmakers in low-cost settings to gather better information
on functional health status.

12

If physical tests are not possible, detailed descriptions

of self-reported health can be collected. A number of
household surveys have been widely implemented to
gather such information. Two examples are the Medical
Outcome Study Short Form Health Survey 36 (SF-36)
and the EuroQuol EQ-5D. The SF-36 measures physical
functioning, mental/emotional functioning, physical role,
emotional role, social functioning, pain and discomfort,
vitality and overall health. This instrument includes
only 36 questions and has been implemented in more
than 50 countries. There is a great deal of literature
assessing its results, and it exists as one pre-set survey
option for use by decision-makers. The EuroQuol EQ-5D
addresses mobility, self-care, usual activities, anxiety and
depression, pain and discomfort and cognition. It has also
been implemented in multiple countries and its results
widely analysed. Since neither of these instruments
include ways to verify the functional ability of the
individual respondent, they are less useful than health
examination surveys. However, they are correspondingly
less expensive to implement.

Human resource inputs

Human resource inputs are an important indicator of
quality and capacity of the health system. The number of
health professionals, and their education and specialty
trainingas well as human resource requirementsare
critical pieces of information for managing the health
system. However, such information is rarely readily
available. Ideally, a well-integrated health information
system will have administrative records that can provide
this informationalthough even these records may not
give a complete picture. Administrative information will

often come from payroll records within a given health

facility. These may not cover all workers who may be
receiving payments from more than one institution
payroll, which is a very common practice in developing
countries. As a result, adding together human resources
that appear on the payrolls of the entire sector tends to
overestimate the number of doctors and nurses. Also,
categorisation of staff often becomes distorted, because
it is common to mix staff with administrative roles and
those whose tasks are more directly related to delivering
health care. This gives a distorted picture of human
resources available to improve the populations health.
In addition, health workers are often discussed as
an artificially homogeneous group, with very little
information about key attributes such as geographical
distribution, types and level of skills, and the balance
between public and private sector employment. Some
countries maintain records of health workers in the
public sector but have little information on the numbers
of private or informal workers.
The best way to collect data on human resources is
to carry out surveys at health facilities with a detailed
labour component using individual identification to
avoid duplication. These surveys can be included in
the routine information system and then providers can
continuously update them. Examples of health facility
surveys are those covered by the International Health
Facility Assessment Network supported by the United
States Agency for International Development (USAID)
and MEASURE Evaluation. These surveys attempt to
evaluate a wider range of resource inputs to health
facilities and include numbers of professionals, their type
and their salaries. Ideally, health facility surveys would
be administered at all facilities in a health system, but
they can also be administered at a representative subset
if available resources prohibit full implementation. It
is important to remember that health facilities are not
just hospitals, so surveys should be carefully designed
to take into account other sites, including community
health clinics and pharmacies, where relevant. While
this approach is the most desirable survey option, there
are some drawbacks. Individual health care providers
may not be captured in a health facility survey; in some
contexts, individual providers make up a significant
portion of overall human resources for health. Also,
health facilities may not have information on the training,
specialty or education levels of their employeesall
of which are key aspects for characterising health
workforce capacity. Finally, individual identification as

Working Paper SeriesNumber 6November 2009

part of a routine health information system can be hard

to achieve even in developed country settings. Finding
ways to account for full-time equivalents, or to otherwise
characterise the actual effort provided by the full
workforce without overcounting, can be challenging.
Censuses can also be a valuable source of information
on workforce geographic diversity, training, specialty and
education levels. For censuses to be useful, they must
include detailed questions about professional categories,
ideally subdividing medical and health professionals into
types such as doctor, nurse or community health worker.
Censuses routinely provide information on age, sex, place
of residence and education level, so the overall data
modifications needed to provide useful information on
human resources are slight. The disadvantage, as always
with censuses, is that they are infrequently collected and
so provide a static picture of human resource capacity,
at best. Historically, detailed professional questions
were not often included in censuses. Their more recent
inclusion is of benefit.

Health financing

In many countries, private expenditure is also a

significant component of health spending. For household
expenditures, governments may want to know whether
the distribution of health costs is equitable across
subpopulations and whether there are subpopulations
unduly affected by high health costs. Surveys are
probably the best means to obtain data on financing of

Household expenditure surveys have been widely

used in many contexts to gather broad financial
information, such as mortgages, consumer spending
and income. The World Economic and Social Survey,
for example, provides objective analyses of pressing
long-term social and economic development issues,
and provides information on the positive and negative
impact of corresponding policies. The Living Standards
Measurement Survey is an example of a finance survey
that helps to gather health-specific data. In a wellfunctioning and comprehensive health information
system, surveys are the most useful way of gathering
information on the actual costs of interventions (through
surveys of providers), and the balance of payments
between out-of-pocket and institutional expenditures
(through surveys of households). This information is
useful for decision-makers interested in how best to
spend limited resources, and in the equitable distribution
of health expenditures across a population. For example,
understanding the true costs of interventions can help
to elucidate barriers to care and eliminate unnecessary
inefficiencies. Understanding the distribution of
expenditures can help decision-makers see whether
certain subpopulations are being unduly burdened
in ways that have increasingly negative impacts; for
example, whether high out-of-pocket expenditures are
bankrupting an underprivileged subset of the population,
and further impoverishing them. Household expenditure
surveys are most helpful when used in conjunction with
the budget and expenditure reports mentioned above,
but even in isolation can be instructive to decisionmakers.
Health facility surveys can also provide information on
health financing and can be used in both public and
private institutions. In addition to understanding the
funds that go into the health system, surveys of health
facilities are able to identify the availability and quantity
of services provided by health facilities, and identify
the costs to both the user and supplier. Health facility
surveys include inventory recording; equipment audit;
infrastructure appraisal; interviews with health workers
or patients; and approximating the costs of services and
secondary outputs, such as patient bed-days. Decision-

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Financial resources are perhaps the most fundamental

inputs in a health care system. Decision-makers
and donors will likely want to know basic costs,
commitments, disbursements and expenditures. It
is additionally useful to understand the relationship
between what is budgeted and what is spent in any
given year. Decision-makers and donors may also find
it useful to understand the proportion of expenditures
made by the government and by households. Much of
this information may be kept by ministries of finance for
monitoring government expenditures overall. Budget
expenditure reports and other financial reports are
therefore valuable sources of this information. In some
instances, countries have set up a system of national
health accounts specifically to track spending. Even
without such a system, expenditure and financial reports
can be used to get a good general picture of where
money is going in terms of health.

health care, both public and private. In many countries,

local or national surveys are widely used to obtain critical
information on health financing. Politicians and local
health officials are more likely to react to their own data.
These data can lead to action and help to guide policy
and the more effective allocation of resources.

13

Health Information Systems Knowledge Hub

14

makers can use this information to understand more

specifically what drives financial costs in health care.
This information can help them to better allocate their
own resources. In addition, providing precise input and
outputs costs is extremely important in supplementing
requests to both national and international funding
sources to make up for shortfalls and provide the
critical capacity that the health system may need to
perform well.

has been implemented. Costs in the private sector

might be harder to track, depending on the dynamics of
the particular health system. However, in these cases,
health facility surveys can be used to collect information
on the actual expenditures by facility for a range of
different inputs, including building costs, personnel,
pharmaceuticals, equipment and medical supplies.
Regardless of whether the facility is public or private,
such surveys can be used to help verify government
administrative information and better understand costs.

Cost-effectiveness

Health facility surveys can also help provide information

on outputs, such as immunisations given, diagnostics
procedures undertaken (number and type), surgeries
performed, bed-days provided and pharmaceuticals
dispensedin both public and private facilities.
This information can be related to the interventions
themselves so that decision-makers have both inputs and
outputs. Judging the actual benefit of these collective
outputs is more difficult and cannot easily be determined
using surveysalthough for highly specific instances,
such as using health examination surveys to judge
blood pressure control provided by pharmaceuticals,
they might be useful. More likely, the benefit of a
given intervention should be taken from the published
literature on clinical trials and intervention effectiveness.
If a common assessment of the benefit is used, decisionmakers can make cost-effectiveness calculations of their
own for a particular context.

Cost-effectiveness is an analysis tool that is used to

compare the cost of a health intervention with the
expected health gains. The interventions that are
included in a cost-effectiveness analysis are those
that use human, financial and other inputs to improve
health by reducing health risks, reducing the severity or
duration of an illness or disability, or preventing death.
For example, to calculate the cost-effectiveness of an
intervention that prevents death from measles either by
immunising a child or by treating the disease, analysts
compare the costs of different interventions that can
achieve that outcome. The result is a cost-effectiveness
ratio, expressed as cost per outcome. The ratio is always
discussed in relative terms, as there is no 'best' or
absolute level of cost-effectiveness (Jamison et al 2006).
Many studies of cost-effectiveness have been published,
and several large efforts have been undertaken to gather
them together in a way that is valuable to decisionmakers in different settings. The WHO CHOICE program
and the Disease Control Priorities Network both
provide cost-effectiveness ratios for several hundred
interventions and, where possible, adjust these ratios
regionally to give decision-makers information that can
be better used within their country.1
Where resources are available to carry out such studies
within a given country, two key pieces of information
must be collected: information on the costs and
information on the outputs. To do so requires leveraging
several different sources of information. For public
spending, budget and expenditure reports can provide
costs for interventions and their related inputs (eg
personnel, facilities, pharmaceuticals). These budget
and expenditure reports are relatively easy to obtain
in countries where a national health accounts system
1

See www.who.int/choice/en/ and www.dcp2.org/page/main/Home.

html for relevant information.

Both these sources of information have drawbacks. Cost

information from either budget and expenditure reports,
or from health facility surveys, may be incomplete or
inexact. Frequently, the more specific the unit of input,
the less likely there is to be specific data widely available
about its true cost at scale. For example, a hospital is
likely to know how much it spent on pharmaceuticals in
a given year, but is less likely to know how much it spent
on a given drug. Also, health facility surveys need to be
designed to cover a representative set of facilities, if not
all of them. This can be challenging, particularly in places
where individual providers are responsible for some
aspects of care or where facilities are very different and
not linked in coherent systems.
When information from budget reports and from health
facility assessments is lacking, it is possible to conduct a
meta-analysis for cost-effectiveness using a combination
of other sources: household surveys, hospital discharge
data, health service registry data and epidemiological
studies. Meta-analyses are not country specific but may

Working Paper SeriesNumber 6November 2009

provide some information on interventions that decisionmakers may wish to assess. For example, in an evaluation
of 16 interventions for neonatal survival, researchers
conducted a meta-analysis using epidemiological studies
and a cost-effectiveness framework (WHO-CHOICE) that
uses unit cost data from countries and other research
estimates (Adam et al 2003). Even without complete
health facility platform and intervention costs, the
researchers were able to provide recommendations on
packages of care, over single interventions, to improve
neonatal survival (Darmstadt et al 2005). From a health
information systems standpoint this may be the least
desirable approach, but could be used strategically when
evaluating priority interventions that are a political and
practical priority, but lack resources to evaluate locally
before urgent decisions are made.

Effective coverage
Effective coverage (Shengelia et al 2005) measures the
percentage of health gain realised through delivering
high-quality interventions to those in need. Effective
coverage captures the concepts of both quality and
utilization. Calculating effective coverage requires surveys
to collect data on three components:
the population in need
the utilization of the intervention within the
population
the quality of that intervention.

Surveys are the primary source of data to address the

second requirementthat is, to determine whether a
specific intervention is being provided in a population
and subsequently used by those who need it. Existing
surveys regularly collect information on the utilization
of health services for family planning, antenatal care,
vaccination coverage and the treatment of childhood
diseases such as diarrhoea. While administrative records
can also be used, surveys tend to give better information
on intervention utilization, particularly as administrative
records suffer from reporting bias (Lim et al 2008).
Finally, quality is the most difficult aspect to measure for
any intervention, which is why many reports of coverage
do not address the effectiveness component. Diagnostic
measurement of baseline versus current state (after
intervention) is the only true standard against which
improvement can be measured, but is unlikely to be
achieved except in clinical settings. Health examination
surveys with followup of participants can be used to
supplement available clinical data. This is one area where
stand-alone research studies of population subgroups
can contribute valuable data.
Even though measurement of effective coverage is
an excellent way for a government to understand
how effective their health systems are, measuring it
usually requires the addition of several metrics into
national surveys to gain an accurate understanding of
the effective coverage of various interventions. Even if
decision-makers were convinced of the need for this
metric, it would require a significant change to the health
information system. This is a good example of how
supply and demand of health information depend on
each other.

Health Information Systems Knowledge Hub

While effective coverage is extremely useful information

for policymakers, there have only been a few attempts
to measure it, most likely because of the perceived
complexity of obtaining all of the information needed
(Murray 2009).

health authorities could identify the proportion of

diagnosed hypertension from self-reported data and
the proportion of undiagnosed blood pressure from the
measurement (ie cases where the respondent was not
previously diagnosed, but their physical measurement at
the time of interview indicated hypertension). Moreover,
the physical examination measurement component can
provide the proportion of controlled and uncontrolled
conditions in the community. This addresses the first
data requirement in measuring effective coverage

Working Paper SeriesNumber 6November 2009

15

To determine the population in need, diagnostic

information is the gold standard measure for defining
whether an individual suffers from a particular condition
or not. Mandatory physical examinations for an entire
population are not possible; instead, health examination
surveys that pair interviews with biological and clinical
measures are often used. If health examination surveys
are not possible, self-reported conditions in interview
surveys are a viable, though not perfect, substitute.
Even a small sample of health examination survey
data can be used to at least validate and adjust selfreported claims. For example, by including a physical
examination measurement component in a survey,

Health services

Health Information Systems Knowledge Hub

Health services information can be captured in many

ways, but the primary goal is to understand the quantity
and quality of inputs and the intermediate outputs that
the health services provide. Health services include
ambulatory care, community and patient outreach, as
well as inpatient and outpatient care. These services
are best captured through health facility assessments
that measure a variety of hospital inputs and outputs.
As mentioned above, health facility assessments can
measure finances and personnel at a given facility. Health
facility assessments can also monitor inputs like medical
supplies, equipment and pharmaceuticals. They can
help to catalogue the number, size and type of facilities
that exist within a health system, as well as those
interventions and services that are actually provided
(eg how many cancer wards exist versus the number of
dental clinics or maternity wards). Finally, health facility
assessments can provide information on outputs, such as
the number of bed-days used, or the number and type
of diagnostics carried out. Additionally, health facility
assessments can incorporate questions that relate to the
quality of service provision at the periphery of the health
system, using questions that cover the type of range of
services provided, how frequently these services are
provided, and to whom.

16

Existing survey instruments can be used as a point

of reference for countries wishing to implement
such measurement strategies. The Service Provision
Assessments by MEASURE DHS focus on five key services:
child health, maternity and newborn care, family
planning, sexually transmitted infections, and other
infectious diseases including HIV. Health service provision
and quality is measured through a series of indicator
questions in these health facility assessments (MEASURE
DHS 2010). A number of surveys in the public domain can
be usefully adapted to individual country contexts in this
regard.
Besides health facility assessments, hospital discharge
data can be potentially useful both to track quality of
outcomes of health services, and to track health service
use. Accurate records, if available, can provide decisionmakers with information about the interventions applied,
the pharmaceuticals prescribed and the procedures
undertaken.
Partial information about health services may be
available from other sources. Some countries will have

health service registries, essentially administrative

records of health facilities. Household surveys can be
used to obtain some information about health service
use, although they lack the ability to provide information
on quality. They can be particularly useful, however,
to reach populations that more likely to use individual
providers than to use institutional health services.

Burden of disease and

comparative risk assessment
The concept of burden of disease brings together
information on mortality, causes of death, morbidity and
functional health status for a population via the use of a
summary health measurethe DALY. The DALY is used to
compare the population-level health impact of different
conditions on overall population health. DALYs have
also been used for comparing the impact of different
health riskssuch as smoking, alcohol and high blood
pressurethrough a related set of methods referred to
as comparative risk assessment. Together, the two are
extremely important pieces of evidence for decisionmakers to understand the main causes of health loss in
their populations. The policy relevance and importance
of the concept of burden of disease is evident in the
myriad national burden of disease studies that have been
undertaken since the original Global Burden of Disease
Study (Murray and Lopez 1997). These studies include
Australia, Brazil, Canada, Chile, China, Colombia, Cuba,
Egypt, Estonia, India, Indonesia, Iran, Italy, Japan, Jordan,
Korea, Lebanon, Malaysia, Mauritius, Mexico, Morocco,
Mozambique, the Netherlands, New Zealand, Oman,
South Africa, Serbia, Sweden, Syria, Thailand, Tunisia,
Turkey, the United Kingdom, the United States and
Zimbabwe.
Ideally, a national health information system should
be able to provide the combination of information
necessary to periodically produce burden of disease and
comparative risk assessment estimates. It is worth noting
that the current Global Burden of Disease Studywhich
is being carried out by a consortium of institutions
led by the Institute for Health Metrics and Evaluation
(IHME) and including to measure disability weights by
assigning comparative 'weights' to various health states
so that estimates can be made about the overall level of
disability. This information is useful for decision-makers,
public health officials and others to understand the
disease burden in their communities. The DALY is an

Working Paper SeriesNumber 6November 2009

important concept in health care accountability because

it measures not only of loss of life, but also reduction in
function.

Health system performance assessment

Measuring the performance of health systems overall is
of utmost importance, and can be done in a variety of
ways. Traditionally, health service performance is defined
as efficiency, or the extent to which the resources used
by the health system achieve the goals that people
value. However, Murray and Evans (2003) describe a
different framework, in which performance is further
defined to include the entire range of activities, from
measuring goal attainment, to the efficiency of input
use, to the way that the health system is functioning.
They note that this framework defines a health system
as all actors, institutions and resources that undertake
health actions, where the primary intent of a health
action is to improve health. Therefore, it is obvious that
health system performance assessments, particularly as a
decision-making tool, are an important metric that draw
on almost every data source available within a country.
A consolidated view of health system performance
assessment emerges as a result of gathering and
analysing information on the essential metrics articulated
above. In each case, decision-makers should adopt an
overall framework that gives equal value to the different
dimensions of a health system and assesses how that
health system is improving the health of a population.
Using reliable, consistent and comprehensive information
to do so will strengthen the evidence base for decisions
and ultimately improve health outcomes.

Health Information Systems Knowledge Hub

Working Paper SeriesNumber 6November 2009

17

Key considerations for country

survey implementation

Health Information Systems Knowledge Hub

Decision-makers and health practitioners will have

limited time and resources to implement broad-based
changes. Therefore, there must be compelling reasons
to stimulate country engagement in the use of health
surveys and the calculation of essential health metrics
to manage the health system. Highlighting instances in
which countries have benefited from becoming more
engaged in survey implementation is one important
way to encourage greater involvement. Another is to
identify ways in which both local and national decisionmakers can mutually benefit from better information in
terms of both accountability and effectiveness. A further
approach is to emphasise the growing international focus
on accountability and evidence-informed planning, from
which countries will benefit by being forward-thinking
and at the forefront of implementation. A strategy
to outline ways in which the greatest value can be
derived from surveys and the information they provide,
while leveraging existing resources and information
streams, will be the single most important component
to achieving effective and sustained engagement by the
policy community in countries.

18

It must also be recognised that each country context is

different. Although the dimensions of health metrics
and evaluation data that have been outlined here
apply to all countries, each country and each health
information system will have its own strengths and
challenges. Therefore, we cannot make a common
set of recommendations for how each country should
collect health data and improve its health information
system overall. Instead, we have tried in this working
paper to articulate the ways in which surveys could
be used to complement existing data environments.
Health decision-makers within each context should make
their own decisions based on the framework we have
proposed and decide the surveys that make the most
sense for their own contexts.
Nonetheless, we believe that decision-makers
everywhere should be aware of three abiding
considerations:
Complement what is currently in place and improve
existing surveys.
Use new technology and multimode surveillance to
contain cost.
Consider the use of surveys to evaluate prevention
programs and policies.

If these considerations are emphasised in the planning

and decision-making around implementing surveys, we
believe that the value of those surveys will be maximised
as part of the health information system overall.

Complement what is currently in

place and improve existing surveys
Building on current country health information systems
rather than uprooting what is there and trying to
fundamentally rebuild is a key strategy for engagement.
There are existing sources of health data in all countries,
and often at least rudimentary information-gathering
routines exist that should be leveraged. A firm sense
of the existing health information system landscape is
needed in each instance to guarantee that no existing
information is cast aside and to ensure that any existing
surveys are used to best advantage. Sometimes, existing
surveyseven if they are not specific to healthcan
have modules about health added to provide useful
information at minimal cost. Often, census data can be
used to generalise updated information from a smaller
sample of household surveys. Understanding the
timing and nature of other efforts from international
organisationssuch as surveys carried out by WHO or
the World Bankcan help to establish benchmarks and
potentially piggyback survey instruments that are already
being used without incurring additional costs in resourcepoor settings. Any new survey effort must be considered
in the context of the health information system as a
whole. If thought of in this way, rather than in a one-off
or ad hoc fashion, then a more complete picture of a
countrys health system can be attained, and existing
data collection efforts sustained and strengthened,
rather than replaced by survey programs.
Additionally, if new surveys are to be implemented, then
every effort should be made to adapt as appropriate
proven surveys that have been used elsewhere. Many
survey instruments with significant research and
analysis behind their effectiveness and results are now
available in the public domain. A number of them have
been mentioned within the descriptions of the key
health metrics areas outlined in Table 1. Together, they
comprise a valuable storehouse of knowledge about
the appropriateness of different survey questions, the
best ways to sample hard-to-reach populations and
how best to present questions in a way that will elicit
useful responses. Local adaptation and calibration

Working Paper SeriesNumber 6November 2009

should be respected, but it should start from a base of

proven technical effectiveness in instrument design and
implementation.

health. It is important to coordinate across partnering

agencies and countries to evaluate the effectiveness of
promising public health strategies. This coordination
could be expanded to further support field evaluation of
evidence-informed strategies.

Use new technology and multimode

surveillance to contain cost
New thinking is required to meet the latest challenges
in data collection, such as the increasing popularity of
mobile cellular telephones, in both the developed world
and resource-poor settings, and the increasing interest of
governments in protecting the privacy and confidentiality
of their citizens health information. In some developed
countries like the United States, the usefulness of the
traditional telephone survey may be nearing an end, and
new ways to conduct surveillance are being explored.
In some resource-poor settings, people who never have
had a landline now have access to cell phones. Innovative
social experiments in areas such as microfinance are
currently under way to use cell phones to connect
suppliers and local markets, and to manage loans and
make transactions. Similar innovative thinking is needed
in the area of health information to creatively adapt to
rapidly changing technology all over the world. New
methods of data collection are required to improve the
accuracy of public health surveillance data. For example,
physical activity has been difficult to measure through
self-reported questionnaire surveys. In the near future,
more accurate instruments that are currently limited to
small samples and specialised studies may be used more
widely in population health surveys, especially when
cost, availability and accuracy are optimised.

Strengthening the capacity of health information systems

refers not only to the scientific process of collecting,
analysing, interpreting and publishing data, but also to
the translation of these data into practical applications.
We have already argued that creating a culture of
information with mutual benefit is critical (IHME and
HIS Knowledge Hub 2009). National and local decisionmakers can together benefit from reduced burden of
diseaseboth in terms of better population health
and lower health system costs. Creating mutual and
compelling incentives for doing so by carefully measuring
the impact of prevention programs and policies could
be a very successful way to create mutual incentives for
high-quality data.

If one of the aims of a health information system is to

better understand how to actively improve the health
of a population, then disease-prevention programs are
of crucial importance. Preventing diseases can often be
far more cost-effective than treating them. While we
believe that each context deserves its own assessment,
it is critical to assess disease prevention and control
programstheir design, implementation, effectiveness,
adoptability and utilitywhich can be done using
surveys. Programs should produce evidence that leads
to effective public health policies and actions to prevent
or reduce the burden of chronic disease while promoting

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Consider the use of surveys to evaluate

prevention programs and policies

19

Survey design and data collection

Once a decision has been made as to what surveys to
implement, there are important implementation issues
to consider. Many different guides for implementing
particular types of surveys are available (International
Health Facility Assessment Network 2008). However,
detailed planning should take place before implementing
a survey, recognising the highly specific characteristics
of the population. The sections below highlight some
particularly important elements to consider.

Sample size and sampling

Health Information Systems Knowledge Hub

Household surveys have varying sample sizes. It is

not straightforward to define the optimal sample size
for a household survey, because this depends on
several issues.

20

Deciding on the key indicators for the survey to monitor

is an important first step. If you are interested in
analysis on a subgroup level, for example among poor
populations or in rural areas, sample sizes need to be
large enough to have representativeness at this level. On
the other hand, if you are only interested in monitoring
national-level indicators, sample sizes can be much
smaller. Decisions on what differences between groups
need to be detected will affect the size of the group
sampled as well. The smaller the differences you need
to detect, either between groups or between years,
the more accurate estimates (with smaller confidence
intervals) are necessary. Sometimes in developing
countries, sample sizes should represent regions or
even the first administrative level: states, provinces,
departments, and so on. These studies are more
expensive due to political pressure to include all areas,
rather than responding to technical needs.
Identifying the variability in the outcome of interest is
important, as well as deciding the sample size. A larger
sample size is required where there is more variability
in the outcome of interest. The required sample size to
observe a given change over time can be calculated using
appropriate statistical techniques. Other decisions to
make at the outset include the minimum difference you
want to detect, the variability in the outcome (possibly
based on a previous estimate), an acceptable probability
of committing the error of seeing a statistically significant
difference when there is none (usually 0.05), and an
acceptable probability of committing the error of not
finding a difference when there actually is one
(usually 0.20).

Survey data can produce information on subgroups of the

population. This can be achieved by stratification, which
means that the population is divided into subgroups
or subpopulations. Sampling is then done separately
for each subpopulation, with no overlap between
each strata. Stratification can be done on the basis of
geography (urbanrural, provinces), socioeconomic
groups or ethnic groups. Stratification is ideal when there
is heterogeneity between different subpopulations, and
is usually used to reduce sampling error and ensure
representation of every subgroup.
A sample is commonly selected from a list of sampling
units such as households. The simplest way of selecting
a sample is by simple random sampling. However, it is
often difficult to obtain a sampling frame due to the
nature of the target sample, because of geographic
obstacles, or if the population is too large. Furthermore,
simple random sampling is sometimes at risk of
oversampling from select subgroups (eg minority groups,
the elderly) due to their accessibility to various surveying
methods. Therefore, many surveys use a more complex
survey design.

Survey costs
Survey costs generally depend on three aspects:
1. Sample design.
2. Data collection and processing.
3. Survey infrastructure.
First, the cost of interviewing a cluster relative to the
cost of interviewing an individual is important. This cost
ratio varies from country to country, depending on the
population density, the level of urbanisation and the
infrastructure of the country. Determining the best way
to minimise the data collection time and effort can be
assessed within the sample design itself, and may
reduce costs.
Second, increasing the efficiency of data collection and
processing may also reduce costs. It is important to keep
in mind that technology is not always more expensive to
use in the field. In general, collecting data in handwritten
form and transferring it to computer is time-consuming
and fraught with input error, frequently meaning that
double-data entry is required. Electronic data capture
using handheld devices and laptops or netbooks may be
both quicker and more accurate. Evidence suggests that

Working Paper SeriesNumber 6November 2009

improved data collection systems, better supervision and

faster analyses save money in the long term.
Third, for one-off surveys, preparing the survey
accounts for most of the total cost. Preparation
includes coordination of survey planning by multiple
government agencies, frame development, sample
design, questionnaire design, printing of questionnaires
and other survey materials, and generating publicity
to notify potential respondents. These costs tend to
be much lower in countries with substantial existing
survey infrastructure than in those with little or
no infrastructure. Frequently, using demonstrated
instruments from other settings can help to cut down on
costs. A number of surveys are now widely available (eg
DHS and BRFSS) and can be adapted fairly easily.

Periodicity and harmonisation of surveys

Quality of survey data

The design and implementation of a survey affects its
data quality. Decision-makers need assurances that
the data are not biased, especially when results are
generalised. The health community has long regarded
response rates as a proxy measure of data quality;
however, this position is not supported by a great deal
ofevidence.

Adequate coverage is when the sampling frame

includes all units of the population of interest. If the
list is incomplete, it can result in frame coverage errors.
Challenges to coverage vary by the survey mode. Usually,
sampling frames for face-to-face surveys are expensive
to develop, while telephone sampling frames can be
challenging because of cellular phones and number
portability.
Adequate sampling is when each element in the
sampling frame has a known and nonzero probability of
selection. This protects against sampling bias and enables
the implementer to quantify sampling error. Again, this
error varies by survey modes. Face-to-face and telephone
surveys have well-developed techniques for sampling.
On the other hand, mail surveys do not have a clear
methodology for within-household selection, and in
many resource-poor settings may not even be possible
to carry out. The implementer cannot control who will
answer the questionnaire once the letter is received.
Nonresponse errors occur due to the inability to obtain
data from selected respondents. There are two aspects
to this error. Unit nonresponse is when the selected
individual refuses to participate in the survey, while
item nonresponse is when the respondent completes
the survey but refuses to answer certain questions.
Again, this error varies by survey mode and questions.
For example, in face-to-face interviews, a respondent
may be less likely to provide personal information on
sexual behaviour to an interviewer of the opposite sex.
However, the same person may be more likely to respond
if the interviewer is of the same sex.
Measurement errors occur when a respondents
answer to a question is inaccurate (departs from the
'true' value). Several factors affect this error. Most
important are question wording and their order in the
questionnaire. Therefore, it is crucial to cognitively
test the questionnaire and pilot the survey before full
implementation. For example, asking 'are you trying

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

Surveys are usually not conducted on a yearly basis,

as this would be too costly, and in many cases it is not
realistic to monitor changes from year to year. A two-year
interval between surveys is more common. Clearly, smallscale surveys will provide data that are more readily
available and perhaps less expensive to collect, but can
have the drawback of yielding less reliable estimates.
Therefore, periodicity and consistency of surveys is of
utmost importance, even if this would mean opting
for a suboptimal survey setup. Harmonisation can be
important in two ways: across different populations
and across time. Within a country, both may be difficult
if local agencies and providers frequently implement
the surveys. However, strong national standardisation
and mutual accountability through public reporting
on a preset timeline can provide an important mutual
incentive for standardisation. If the reporting is done
regularly (eg every several years), continuity among the
surveys themselves may be important to preserve
remembering to adapt surveys to meet emerging needs.

The quality of a survey is determined by five

components: coverage, sampling, nonresponse,
measurements and data processing. Errors could
occur with each component. Therefore, using only
one component to determine quality such as only
using high-density lipoprotein as a measure of good
health and ignoring low-density lipoprotein, blood
pressure or diabetes, would lead to inaccurate quality
measurementresults.

21

to lose weight?' or 'weight loss is important for your

health; are you trying to lose weight?' will yield different
estimates for weight loss attempts. The survey mode
has significant implications for measurement error
(interviewer versus self-administered). This error will be
affected by the interviewer stimuli and/or the manner
in which the survey questions are conveyed to the
respondent and the way that responses are recorded.
Data processing errors occur during data management,
editing and recoding. Sometimes, errors are made during
imputations of certain missing items or responses.
Finally, errors could be made when calculating final
weights or post-stratification adjustments. Hence,
there is a need to have systems in place during a survey
operation for quality assurance and control.

This last point is the most relevant, because it provides a

benchmark with other surveys or census data.
Beyond the technological problems or future directions
of the management of health information systems, it
is important to remember current survey methods still
have problems of completeness, missing values and the
differential bias of collected data from routine registries.
Health information systems, through standardisation of
protocols and survey instruments, could help reduce
this problem.

Surveys should provide information on their data

quality to address these potential concerns explained
above. Ideally, all surveys should provide an operational
and users guide, as well as a data quality report. The
operational and users guide should include full details
of the survey design, target population, sampling,
instruments and methods used. The guide should contain
sections describing the following information:
questionnaire construction and testing (cognitive
andpilots)
interviewers' and supervisors' training and protocols
for monitoring data collection
sampling frame (how data were collected and
maintained)

Health Information Systems Knowledge Hub

sample selection
data collection procedures

22

Working Paper SeriesNumber 6November 2009

data management, including details on data editing,

correction process, and imputations techniques
ifapplied
details on weighting and post-stratification
methodology, if applicable.
The data quality report should provide selected statistical
indicators of data quality assurance and control. The
following should be fully described:
outcome measures, including response rates
disposition codes
selection biases with respect to key variables such as
sex, age or race/ethnicity.

Conclusion
Health surveys should be an integral part of national
health information systems. While they can provide
very important data as part of the health information
system, they are not a panacea to alleviate gaps in health
information. Therefore, countries should selectively and
strategically consider how to best use surveys in their
own context, and pay equal attention to data collection,
processing and dissemination, particularly considering
the metrics that have been presented here as essential to
understanding the functioning of any health system. The
value of the information that a health information system
produces depends on both the type of information and
the degree to which it is used. The same is also true for
surveys. In addition, for summary metrics such as burden
of disease, which use a combination of data gathered
from several metrics, value is contingent upon several
well-functioning data collection systems and does not
rely on just one source for quality.
Each essential metric or indicator used by a country to
monitor their health situation should be linked with one
or more suitable data source (see Table 1). Sometimes
there will be only one data collection method for a
given indicator. The purpose should not be to establish a
'gold standard' for each metric and data source. Rather,
considering limited financial resources, as well as the
complexity of the metric and data source, comparative
ease of integration into a health system and data validity,
the challenge for countries is to find which combination
of indicators and data sources produces the best quality
results. Typically, data for particular indicators can be
generated from more than one source, such as the use of
vital statistics or household surveys to measure infant or
maternal mortality.

Surveys are a key component of strengthening health

information systems, and the role that they can play,
particularly in strengthening primary microdata, is
pivotal to generating useful information and adding to
the comprehensiveness of health information systems.
Surveys will only prove useful, however, if key elements
and characteristics are followed; they must have
strong mutual support from decision-makers, health
professionals and those directly involved with survey
implementation. They must be part of a culture of
information that provides mutual incentives for ensuring
timely and high-quality data. Surveys must also fit
into the national culture, such as whether or not their
application requires the use of cell phones, home visits
or health examinations. They must collect information
that is directly relevant to the health information
system overall and that has explicit value for those
responsible for managing and implementing the health
system in a country. However, surveys should never be
seen as the sole provider of information on health in a
country. Indeed, as demonstrated in the explanation of
essential health system metrics in this working paper,
sometimes surveys may not be the optimal source of
information and may not even be possible in some
cases. However, if decision-makers think strategically
about the essential health metrics articulated here, and
prioritise their resources to gain the most value from
various information sources, including surveys, then they
will have a much more complete and useful picture of
how their health system performs. This will increase the
evidence base for decision-making, ultimately leading to
better health outcomes.

Working Paper SeriesNumber 6November 2009

Health Information Systems Knowledge Hub

In such cases, given the criteria above, one method

may be judged to be more cost-effective than another.
Alternatively, each measurement strategy may have
relative advantages and disadvantages. For example,
data on child mortality or immunisation coverage can be
generated from health service statistics, as well as from
household surveys. However, the use of different sources
generally results in different values for the indicators.
Each source has its advantages and limitations. Service
records are available on a continuous basis but are
subject to bias because they cover only the population
using health facilities. Population surveys are considered
to be more representative but they are time-consuming
and expensive. As a result, they are generally conducted
only occasionally and generate results often spanning
several years rather than the immediate past.

The reconciliation of estimates arising from different

sources and measurement techniques is necessary
when there is no single best data source. Several health
statistics can be obtained from multiple data sources,
and reconciliation of the data is needed to obtain the
best estimate of rates and trends. Population surveys
generally do not generate yearly data or provide
information for small geographical areas. Although
limited to those who take advantage of formal health
facilities, health facility data can provide useful
information on small populations if special investments
are made to ascertain data quality and completeness.
Population survey results may need to be used to adjust
the rates and trends observed from data on health
facilities.

23

References
Adam T, Evans D and Koopmanschap M (2003). Costeffectiveness analysis: can we reduce variability in costing
methods? International Journal of Technology Assessment in
Health Care 19:407420.
Adjuik M, Smith T, Clark S et al (2006). Cause-specific mortality
rates in sub-Saharan Africa and Bangladesh. Bulletin of the
World Health Organization 84:181188.
Darmstadt GL, Bhutta Z, Cousens S et al (2005). Evidencebased, cost-effective interventions: how many newborn babies
can we save? The Lancet 365(9463):977988.
Editorial (2006). The crisis in human resources for health. The
Lancet 367(9517):1117.
Evans D, Adams T, Tan-Torres Edejer T et al (2005). Time
to reassess strategies for improving health in developing
countries. BMJ 331(7525):11331136.
Friedman CP and Wyatt JC (1997). Evaluation Methods in
Medical Informatics, Springer, United Kingdom.
Gakidou E, Hogan M and Lopez AD (2004). Adult
mortality: time for a reappraisal. International Journal of
Epidemiology33:18.
Gauld R and Goldfish S (2006). Dangerous Enthusiasm:
E-Government, Computer Failure and Information Systems
Development, Otago University Press, Dunedin.
Heeks R (2002). Information systems and developing countries:
failure, success, and local improvisations. The Information
Society 18(2):101112.
IHME and HIS Knowledge Hub (2009). Improving the quality
and use of health information: essential strategic issues. Health
Information Systems Knowledge Hub Working Paper Series 5.

MEASURE DHS (2010). SPA Overview, MEASURE DHS.

www.measuredhs.com/aboutsurveys/spa/start.cfm (Accessed
29 April 2010).
Mokdad A (2009). The behavioral risk factors surveillance
system: past, present, and future. Annual Review of Public
Health 30:4354.
Murray CJL (2007). Towards good practice for health statistics:
lessons from the Millennium Development Goal health
indicators. The Lancet 369(9564):862873.
Murray CJL, Lopez AD, Feehan D et al (2007). Validation of the
symptom pattern method for analyzing verbal autopsy data.
PLoS Medicine 4(11):e327.
Murray CJL and Lopez AD (1997). Regional patterns of disabilityfree life expectancy and disability-adjusted life expectancy:
results of the Global Burden of Disease Study. The Lancet
349:13471352.
Murray CJL and Lopez AD (2006). The Global Burden of
Disease, World Health Organization, Harvard School of Public
Health, World Bank, Geneva.
Murray CJL and Evans D (2003). Health Systems Performance
Assessment: Debates, Methods and Empiricism, World Health
Organization, Geneva.
Population Health Metrics Research Consortium Project (2010).
IHME, Population Health Metrics Research Consortium Project.
www.healthmetricsandevaluation.org/what/areas/outcomes/
gc13.html (Accessed 1 March 2010).
Preston SH, Heuveline P and Guillot M (2001). Demography:
Measuring and Modeling Population Processes, Blackwell
Publishers, Oxford.
Shengelia B, Tandon A, Adams O et al (2005). Access,
utilization, quality, and effective coverage: an integrated
conceptual framework and measurement strategy. Social
Science and Medicine 61(1):97109.

Health Information Systems Knowledge Hub

International Health Facility Assessment Network (2008).

Flowchart of Steps to Conduct a Health Facility Assessment
(MS-08-28), MEASURE Evaluation, Chapel Hill.

Mathers C, Sadana R, Saloman J et al (2001). Health life

expectancy in 191 countries. The Lancet 357(9269):16851691.

Jamison DT, Breman J, Measham AR et al (eds) (2006). Disease

Control Priorities in Developing Countries, 2nd edn, Oxford
University Press, Oxford and New York.

24

Working Paper SeriesNumber 6November 2009

Kuhn KA and Giuse DA (2001). From hospital information

systems to health information systems. Problems, challenges,
perspectives. Methods of Information in Medicine
40(4):27587.
Lim S, Stein D, Charrow A et al (2008). Tracking progress
towards universal childhood immunisation and the impact
of global initiatives: a systematic analysis of three-dose
diphtheria, tetanus, and pertussis immunisation coverage. The
Lancet 13(372):20312046.
Littlejohns P, Wyatt JC and Garvican L (2003). Evaluating
computerised health information systems: hard lessons still to
be learn. BMJ 326(7394):860863.

Van der Meijden MJ, Tange HJ and Pasman JT (2003).

Determinants of success of inpatient clinical information
systems: a literature review. Journal of the American Medical
Informatics Association 10:235243.
WHO (2002). Revised Global Burden of Disease (GBD) 2002
Estimates, World Health Organization, Geneva.
WHO and Health Information Systems Knowledge Hub (2009).
Improving the quality and use of birth, death and cause of
death information: guidance for a standards-based review of
country practices. Health Information Systems Knowledge Hub
Working Paper Series 1.
Yang G, Rao C, Ma J et al (2006). Validation of verbal autopsy
procedures for adult deaths in China. International Journal of
Epidemiology 35(3):741748.

The Knowledge Hubs for Health Initiative

The Health Information Systems Knowledge
Hub is one of four hubs established by
AusAID in 2008 as part of the Australian
Governments commitment to meeting
the Millennium Development Goals and
improving health in the Asia and Pacific
regions. All four hubs share the common
goal of expanding the expertise and
knowledge base to help inform and guide
health policy.
The Knowledge Hubs are funded by
AusAIDs Strategic Partnership for
HealthInitiative.

Health Information Systems Knowledge Hub

The University of Queensland
Aims to facilitate the development and integration of health
information systems into the broader health system strengthening
agenda, and increase local capacity to ensure that cost-effective,
timely, reliable and relevant information is available. The Health
Information Systems Knowledge Hub also aims to better inform
health information systems policies across Asia and the Pacific.
www.uq.edu.au/hishub

Human Resources for Health Knowledge Hub

The University of New South Wales
Aims to contribute to the quality and effectiveness of Australias
engagement in the health sector in the AsiaPacific region by
developing innovative policy options for strengthening human
resources for health systems. The hub supports regional, national
and international partners to develop effective evidence-informed
national policy-making in the field of human resources for health.
www.hrhhub.unsw.edu.au

Health Finance and Health Policy Knowledge Hub

The Nossal Institute for Global Health
(University of Melbourne)
Aims to support regional, national and international partners
to develop effective evidence-informed national policy-making,
particularly in the field of health finance and health systems. Key
thematic areas for this hub include comparative analysis of health
finance interventions and health system outcomes; the role of
non-state providers of health care; and health policy development
in the Pacific.
www.ni.unimelb.edu.au

Compass: Womens and Childrens Health Knowledge Hub

Compass is a partnership between the Centre for International
Child Health, The University of Melbourne, Menzies School
of Health Research and Burnet Institutes Centre for
InternationalHealth.
Aims to enhance the quality and effectiveness of women's and
childrens health interventions and focuses on supporting the
Millennium Development Goals 4 and 5improved maternal
and child health, and universal access to reproductive health. Key
thematic areas for this hub include regional strategies for child
survival; strengthening health systems for maternal and newborn
health; adolescent reproductive health; and nutrition.
www.wchknowledgehub.com.au

A strategic partnerships initiative funded by the Australian Agency for International Development

HUMAN RESOURCES FOR

HEALTH KNOWLEDGE HUB

The Nossal Institute

for Global Health