Data Quality

PROGRAMMATIC M&E LFA TRAINING

LFA TRAINING 2019/2020

GENEVA, SWITZERLAND
 Data Quality

Presentation outline:

1 Data Quality and Importance to the Global Fund

2 Approach to data quality assessment

3 Planning and Implementation

4 Expected outputs

Use of Targeted DQR results

5

Case study – Review & Discussion

6

2
1 Data Quality and Importance to the Global Fund

What is good quality data ?

• Data fit for purpose - sound data for planning and setting priorities
• Complete – indicator data elements
• Reporting completeness
• Timely reporting
• Accurate
• Reliable

3
1 Data Quality and Importance to the Global Fund

➢ Data Quality is a key risk for the Global Fund

➢ Quality data is required for sound decision making program planning, investment decisions,
monitoring, program performance, quality improvement

➢ Health facility data is the primary source for assessing health sector performance. The quality of
routine data reported by health facilities should be assessed regularly and required investments
should be made to ensure data is reliable and useable

➢ Countries with poor and very poor data quality will receive regular data quality reviews to closely
monitor improvements over time

➢ Data quality reviews will be less frequent in countries with good data quality

4
1 Why we need good quality data ( country level and GF)

Planning / Investments Program Improvement Monitoring and Evaluation

• NSP • Monitoring of implementation • Quality

• Concept note • Decision making • Effectiveness

• Prioritization • Modification • Efficiency, etc.

• Targeting • Annual funding decision • Results reporting

• Finding missing cases, KPs, etc. • Reprograming • KPI, etc.

• Allocation

• Strategic investment decisions

1 Data Use for Action and Improvement (DUFAI) Framework

Country processes
1. Investments in
country data
2. Program
systems and 4. Evaluations
monitoring 3. Systematic
analytical
data analysis &
capacities
synthesis

5. Data use through ongoing dialogue, action and improvement

Global Fund entry points

Improve data availability & quality Analytics and data use Comprehensively
Support partners in development of Build in-country analytical evaluate for learning
normative tools and guidance, and capacity, Strengthen data and accountability
dissemination harmonized tools and analysis and program GF-led evaluations in
process to countries. reviews: focused countries:
Invest in HMIS / DHIS through regional Technical assistance pool: Thematic reviews:
and global software development and TA
network

Coordination with partners

6
1 Component 2: Program monitoring

Purpose:

To track program performance and to assist in effective Management, and timely

decision-making through systematic collection of data alongside program implementation

• Performance

• M&E System Strengthening Activities

• Program Quality

• Data Quality

7
1 Options for Programmatic and M&E Assurance and Service Providers

Assurance Main service provider Assurance Service provider

Review of data systems (community/ Country led (TA as needed) identified Partners review Partners
facility) service provider

Program quality/ data quality spot LFA Country evaluations GF-led with service provider in
checks focused countries
Country led with TA if needed
Health facility assessment Country led (TA as needed)/ LFA/ Thematic reviews
(national/targeted) identified service provider Service provider

Data quality reviews Country led (TA as needed/ LFA/ Prospective Country Evaluations
identified service provider Service provider
(national/targeted)
Review of Laboratory systems Country led (TA as needed)/ LFA/ Population-based surveys Country led (TA as needed) or
identified service provider partner contracted service provider

Routine programmatic analysis Country led (TA as needed) or

identified service provider in some
scenarios The LFA service providers should have technical
Program reviews Country led (TA as needed) with skills and competencies
support of identified service provider
in some scenarios
8
1 Role of LFA in Programmatic and M&E Assurance

Programmatic and M&E Assurance

Associated LFA Service (2018 LFA Services list)
options
1.Review of data systems (community/facility) M&E system assessment
− Program and/or data quality spot checks;
2. Program and/or data quality spot checks
− Joint programmatic, financial and supply chain spot checks
3. Health facility assessments Targeted HFA
4. Data quality reviews Targeted DQR
Review of medical lab systems/services, including lab-related supply
5. Review of laboratory system
chain
6. Program reviews n/a
7. Partner reviews n/a
8. Routine programmatic analysis n/a
9. Population -based surveys n/a
10. Country evaluations n/a
11. Thematic reviews n/a
12. Prospective Country Evaluations n/a
13. Community monitoring n/a
PU/DR or PU: Verification of Programmatic Performance; Review of
9
grant documents (PF, M&E plan)
2 3
DQR Tools
WHO DQR
Framework and
Metrics
Modules 1, 2 & 3

4 5
9 7

8
2
7
3

0
5
6 1

4
2 Harmonized Data Quality Assessment Tool: WHO Data Quality Review (DQR)

❖ Multi-pronged and harmonized approach for data quality assurance: DQR framework is
a collaborative effort of WHO, GF, GAVI, JSI & MEASURE Evaluation

❖ Data quality analysis of a set of core tracer indicators across multiple program areas & a
system assessment

❖ Permits the Global Fund to know that the routine data have undergone a known minimum
level of scrutiny which lends credibility and confidence in the data

11
2 Approach to data quality: WHO Data Quality Review (DQR) Framework

➢ To assess facility-reported data, including routine monitoring of data from priority health
programs;
Purpose
➢ Provides risk assurance for service delivered at health facilities

• Harmonized and holistic assessment of the quality of data collected from health facilities
• Quantifies problems of data completeness, timeliness and accuracy according to program areas;
• Identifies weaknesses in the data management system
• Monitors performance of data quality over time
• Results in a data quality improvement plan to address weaknesses in data

Scope Recommended frequency

National DQR: nationally representative sample ➢ National DQR: within 2 years if the ‘data quality’ rating is poor/ very poor and
every 3-4 years, if the ‘data quality’ rating is moderate/good

Targeted DQR: 20-40 sites, with a geographic area ➢ Targeted DQR: based on programmatic risks and context
generally, not statistically representative

12
2 Data Quality Review Toolkit: 3 Components

DQR tool has 3 components implemented together in a national DQR; targeted DQR implement all or some
with a smaller and more targeted sample size (~20-40), and in a geographic area in a country

Data Verification - verification of indicator values sent from

health facility to the next level of reporting(i.e., - Conducted at Facility &
district), and an evaluation of completeness and timeliness of District level
reporting and required data.
DQR

System assessment - evaluation of the extent to which

- Conducted at Facility &
critical elements of the reporting system and data-use adhere District level
to a minimum set of acceptable standards.

Desk review - examines data reported to national level; 1) completeness of reporting, 2) Examines a core set of tracer
internal consistency, 3) external indicators selected across
quality of aggregate reported data for programme indicators consistency of data, 4) external program areas in relation to
is examined using 4 standardized data quality metrics: comparisons these dimension

➢ Questionnaires for Data Verification, System Assessment

➢ LFA Excel tool for data entry and analysis
2 Data Quality Review Metrics Tracked at The Global Fund

1. Completeness of facility reporting per selected program indicator

2. Timeliness of facility reporting per selected program indicator

3. Data Accuracy

i. Verification Factor per indicator

ii. % Exact Match, % Over & % Under reporting per indicator
2 Reporting Performance- Completeness & Timeliness

Applies to health-facility reporting to the next level.

➢ Example: Health Facility to districts, and district reporting to the regional or provincial levels
o Completeness of indicator data: minimum set of variables
▪ Measures whether the health facilities have included information on each of the
selected indicators in their monthly reporting form.

o Completeness of facility reporting:

▪ Assessed by measuring whether all health facilities that are supposed to report
actually do so

o Timeliness of facility reporting:

▪ Assessed by measuring whether all health facilities that submitted reports did so
before a pre-defined deadline

.
2 Reporting Performance: Accuracy

o Focus
Consistency of reported data and original records
o Process:
➢Data Verification
➢Requires collection of primary data from health facilities
➢Assess the reporting accuracy for selected indicators through the review of source
documents in health facilities, and in the national database/HMIS
➢Yields a verification factor: the degree of disparity between the reported number and
recounted number
o Calculation
➢Recounted number of service outputs recorded in source documents at health facilities,
divided by the number of service outputs reported through the reporting system for selected
indicators
2

WHO DQR ➢ A focused review and assessment of data quality with a smaller
Targeted and more targeted sample
Data Quality ➢ Country teams can select any or all three components of the DQR
Review to implement, based on objectives of the assessment
Modules 1, 2
&3 1. Data Verification
2. M&E Systems Assessment
3. Desk Review
➢ Targeted DQR may focus on 1 or more diseases (HIV, TB, malaria)
➢ Conducted by LFA
➢ ~ 20-40 sites
2 When is a Targeted DQR Implemented ?

• When a national DQR did not cover a program area of concern (e.g. community
services)
• To assess the quality of reported data for specific program areas supported by
GF grants
• When there is an identified data quality risk & country context
• To assess the strengths and limitations of data prior to using it for planning and
program management
• As requested by Country Team
2 DQR Options and Implementation
The data quality assessment approach differentiates assessment methods/activities across the three country
categories:

Country Category
Assessment Approach

High Impact Targeted DQR can be conducted anytime, based on risk and context
countries

Core countries Targeted DQR can be conducted anytime, based on risk and context

Focus countries Ad hoc based on risks

19
3 Planning Targeted DQR

1 MECA is available to support Country Team PHME Specialist during the planning, implementation and to review report

2 Planning for programmatic assurance activities starts each year in August during the annual Assurance planning process

3 PHME Coordinates with the LFA to define specific objectives, budget and LoEs for implementing DQR.

4 Targeted DQR : Guidance, Generic scope of work, tools, planning and reporting templates available
3 Planning a Targeted DQR
• Customized Terms of Reference to document and agree with GF Country Team on the objectives, scope and
details of the assessment

• Planning Template – to document and finalize with the GF Country Team all planning details for the assessment
✓ LFA plans to implement the tasks and deliverables; Country team approval prior to the start of activities
✓ Coordination of planned activities with the country stakeholders: the Ministry of Health, Principal Recipients and with
the specific focal points in country

Sampling Approach – Purposive /Convenience

✓ Sites selected for Data Verification and Systems Assessment should be agreed with the Country team [20-40sites]
✓ Ensure sites selected provide health services for the selected indicator and has source documents
✓ LoEs & Budget approval by the Country team prior to start of activities

• Facility & District Data Verification, M&E Systems Assessments

✓ DQR Guidance
✓ User Guide to the WHO Data Quality Review Toolkit
✓ Questionnaires for data verification and System Assessment
✓ Excel Sheet for Data entry and Analysis
21
✓ Reporting Template
3 Planning a Targeted DQR
Agree with the CT on scope and details of the assessment, and complete the planning template

What is needed

1. Statement of Work/Terms of Reference incl. Budget

2. WHO DQR Guidance –Framework & Metrics
3. GF Planning Template
4. DQR Tools: Questionnaires,
5. Data Entry Excel Sheets for data capture and analysis

6. Reporting template-The primary report to be submitted for the Targeted DQR assessment,
containing key results and recommendations from all three components

22
3 WHO Recommended Program Indicators to Assess for Data Quality

➢ Antenatal care 1st visit ➢ Number (%) of pregnant women who attended at least once during their
(ANC1) pregnancy

➢ DTP3/Penta3 ➢ Number (%) of children < 1 year receiving three doses of DTP/Penta vaccine

➢ Currently on ART ➢ Number and % of people living with HIV who are currently receiving ART

➢ Notified cases of all form ➢ Number (%) of all forms of TB cases (i.e. bacteriologically confirmed plus
of TB clinically diagnosed) reported to the national health authority in the past year
(new and relapse)

➢ Number (%) of all suspected malaria cases that were confirmed by

➢ Confirmed malaria cases
microscopy or RDT

23
3 Additional DQR Indicators for Data Quality

HIV

• People living with HIV who have been diagnosed

• Number (%) of people living with HIV who have been diagnosed
• HIV care coverage: Number (%) of people living with HIV who are receiving HIV care
(including ART)
• PMTCT ART coverage: Number (%) of HIV-positive pregnant women who received ART during
pregnancy
• ART retention: Number (%) of people living with HIV and on ART who are retained on ART 12
months after initiation (and 24, 36, 48, and 60 months)
• Viral suppression: Number (%) of people on ART who have suppressed viral load

24
3 Additional DQR Indicators for Data Quality

TB
• Notified cases of all forms of TB
• TB treatment success rate
• Second-line TB treatment success rate

TB-HIV
Proportion of registered new and relapse TB patients with documented HIV status
• Number of new and relapse TB patients who had an HIV test result recorded in the TB register,
expressed as a percentage of the number registered during the reporting period
• Proportion of HIV-positive new and relapse TB patients on ART during TB treatment
• Number of HIV-positive new and relapse TB patients who received ART during TB treatment
expressed as a percentage of those registered during the reporting period
3 Additional DQR Indicators for Data Quality

MALARIA
• Malaria diagnostic testing rate: Number (%) of all suspected malaria cases that received a
parasitological test [= Number tested / (number tested + number presumed)]

• Confirmed malaria cases receiving treatment: Number (%) of confirmed malaria cases treated that
received first-line antimalarial treatment according to national policy at public sector facilities

• Malaria cases (suspected and confirmed) receiving treatment: Number (%) of malaria cases
(presumed and confirmed) that received first-line antimalarial treatment

• IPTp3 : Number (%) of pregnant women attending antenatal clinics who received three or more
doses of intermittent preventive treatment for malaria

26
3 Steps to Implement Data Verification and System Assessment

1. Planning including coordination with PR, MoH & in country stakeholders

2. Sampling approach: objective is not a representative sample but rather a focused review

3. Adaptation of survey tools

4. Field work / Data collection

5. Quality control

6. Data management, analysis, interpretation

7. Report writing including prioritized recommendations

4 Data Verification(DV) & System Assessment(SA) Outputs
 Outputs of Targeted DQR - Data Verification
✓ Data flow per indicator
✓ Completeness of Indicator Data
✓ Reporting Completeness
✓ Reporting Timeliness
✓ Accuracy: Verification Factor per indicator, over and under reporting

System Assessment Tracer indicators = minimum set of acceptable standards

1. % Availability of guidelines at facility level

2. % Availability of trained staff
3. % Availability of tally sheets, registers and reports in forms in last 6 months

4. Receipt of supervision written feedback including on data quality

5. Analysis and use of data
4 Targeted DQR Reporting

• Template: A structure to report the results of the assessment including analytical outputs and prioritized
recommendations
New! Include the Data Flow per indicator
➢ Facility Level Data Quality Metrics Accuracy Good
Very poor Poor Moderate
+/-5%
(1) Verification Factor(VF) key >20% +/-11% to 20% + /- 6% to 10%

Indicator Verification Ratio of recounted number

New! Include Rating per Program indicator Factor Enter Result
of events from source
documents divided by the
reported number of events
from the HMIS
(2) Facility /District Level Reporting Performance
1. Exact % of facilities for which Enter Result
match source data exactly match
✓Completeness of Indicator data reported data

✓Completeness of Facility Reporting 2. Over- % of facilities that over- Enter Result

reporting report by more than 10% (VF
✓Timeliness of Facility Reporting < 0.90)

3.Under- % of facilities that under- Enter Result

reporting report by more than 10% (VF
> 1.10)

29
4 Facility Level Data quality Metrics: Summary facility level Data Verification results by indicator

S/n Disease Name of Indicator Verification factor (ratio of recounted/reported)

component Facility

Reported Recounted Verification Rating per Indicator

value for 3 value for 3 Factor Very
months months poor/Poor/Moderate/Good

[1]

For TB indicators only measured by Quarter to delete the Month 1 and Month 2 columns and change the Month 3 name to the quarter name.

30
4 Detailed Facility Level Data Verification Results

Month 1: Month 2: Month 3:

Name of
Service
No District
Delivery
Point(SDP)
Verificati Weighted Reason for Verification Weighted Reason for Verification Weighted Reason for
on Factor Error variance Factor Error variance Factor Error variance

#of SDPs Total

assessed

31
4 Summary Facility Level Reporting Performance by Indicator

Facility level Data Quality Metrics

S/n Disease Indicator

component/Program area 1) Timeliness of facility 2) Completeness of 3) Completeness of
reporting: facility reporting: Indicator data:

% of submitted facility % of expected facility % reports that were

monthly reports monthly reports that were complete (out of all
received on time actually received received)

32
5 How the Global Fund uses Data Quality Results

• Verification of reported results for grant management

• Assurance purposes

• Implement actionable, recommendations for data quality improvements

• To monitor status of data systems and for improvements

• To monitor the quality of data at facility, district and national level

• To track Data Quality indicators and ratings for all countries(HI/Core)

• For the M&E System Profile-DASHBOARD

(used for country portfolio reviews, etc.)
5 Tracking Data Quality Results
DQR DQR
Country HMIS Data Final DQR Timeliness Completene
Current Risk DQR Accuracy(VF) Source /Status
Quality Score of facility ss of facility
reporting reporting
2.77 93% 92% 83.75% [78.0%, 100% Malaria and TB indicators:1.00 Targeted DQR 2018
A Malaria: 89.5%]
2.33 82% not reported not reported not reported HIV ART VF = 1 Targeted DQR 2017
B
2.26 49% 51% 20% 82% Accuracy : 0.998 National DQR 2017
C
1.67 78% not reported not reported not reported VF: 1 Targeted DQR 2016
D
1 97% Rating 4: very 100% 100% VF for 2 HIV indicators:0.99 and 1.01 Targeted DQR 2017
E good from report Score= 1.0

3.33 32% 99% 98% 100% VF range 0.99-2.98 National DQA-ART 2018
F
3.00 83% 97% 94% 100% Malaria cases tested: 0.94 Targeted DRQ 2019
G Suspect cases 0.98
Suspect cases tested: 0.91
Malaria cases confirmed& ACT Prescription:
1.04
19 National Malaria cases 1.05
2 76%
H DQR not reported not reported not reported TB: 0.99
Malaria 0.83
National DQR 2018

HIV: 0.97 Score =0.93

2.65 86% not reported not reported not reported PLHIV on ART VF = 0.876 National DQA ART 2018
I
3.67 60% not reported not reported not reported Malaria : suspect cases tested -recount / Targeted DQR 2016
J reported -verification
Site 1) 411/248[1.65]; site 2)261/262
[0.99; site 3) 1240/413[3.0]
3.00 84% 97% not reported for HIV/TB/Malaria: Data accuracy: TB cases National DQR 2018
K HIV/TB/Malaria 91.1% ;99.5%; VF = 1.01; Malaria VF = 1.08;
Indicators 99.2% Current on ART =0.95
Average: 1.01
5 Data Quality Source: Decision tree
Has a National DQR been implemented within
the last 3 years?

Yes No

For the 3 diseases, are there results for: Is there HMIS data for No Has there been a targeted
completeness and timeliness of DQR implemented in the
1) completeness of facility reporting facility reporting for the 3 diseases last 2 years?
2) timeliness of facility reporting within the last 6 -12 months?
3) accuracy and its distribution?

Yes No

Yes No Yes

4) Use this Consult

1) Preferred Are there results for Are there results for Are there no results Country
DQ source only completeness only completeness or for completeness 3) Use this DQ Source Team
and Timeliness of timeliness of facility and timeliness of DQ Source
facility reporting? reporting? facility reporting?

Yes Yes Yes

Check Check
2) Use this availability of availability of
DQ Source HMIS Source HMIS Source
6 Group Work, Review & Discussion

Targeted DQR Case Study

i) Completed Targeted DQR Planning template

• Review the information on the template

• Discuss the tools and process for implementing this targeted DQR
• What are critical outputs envisaged?

ii) Presentation to the larger group

36