SYSTEMATIC

REVIEW
Erna Rochmawati, PhD
 Number of review paper in PubMed

9820
8345 8537
7845

2018 2019 2020 2021

Year Number
WHAT IS SYSTEMATIC LITERATURE REVIEW?

¡ A review that can be served as background for an empirical study or a stand-

alone piece (Templier and Paré 2015).
BENEFITS OF CONDUCTING A REVIEW

¡ Assessment of the current state of research on a topic.

¡ Identification of the experts on a particular topic

¡ Identification of key questions about a topic that need further research.

¡ Determination of methodologies used in past studies of the same or similar topics

¡ a means to finding a robust and sensible answer to a focused research question, but not as an end in
themselves.

¡ Provision of clear and comprehensive overview of available evidence on a given topic.

¡ Identification questions for which the available evidence provide clear answers and thus for which further
research is not necessary
Types of review based on the purpose

DESCRIBE TEST EXTEND CRITIQUE

 Types of review based on the purpose (Xiao & Watson, 2017)

Purpose Literature Review Methodology Description

review type description
Describe All literature Narrative Green et al 2001 The authors do not attempt to
types review summarize the entire scope of
literature, but rather identify key
topics related to the research
question and provide a descriptive
account of the evidence in support of
their conclusion.

Textual Popay et al. 2006; Lucas et Authors categorize the findings into
narrative al. 2007 subgroups to organize the literature
 Types of review based on the purpose (Xiao & Watson, 2017)

Purpose Literature Review Methodology Description

review type description
Describe All literature Meta Sandelowsky et al 2007 Authors utilize metasummary
types summary techniques to extract themes.
Meta- Greenhalgh et al 2005 Consideration of the overarching
narrative research tradition affection the
conceptualization of innovation
diffusion. Using systematic review
methodology to describe the literature
regarding diffusion in health service

Scoping Arskey and O’Malley 2005 To systematically search, extract and

review map broad evidence of interest
 Types of review based on the purpose (Xiao & Watson, 2017)

Purpose Literature Review Methodology Description

review type description
Test Quantitative Meta- Glass 1976 To extract and weight evidence
analysis

Mixed Bayesian Spiegelhalter et al 1999; To understand factors behind a

meta- Sutton & Abrams 2001 particular interest, calculating prior
analysis and posterior probability

Realist Pawson et al 2005 “what works for whom under what

review conditions” when it comes to group-
based physical activity interventions.
The authors separate each portion
into distance tables and comment on
the effectiveness of this type of
intervention.
 Types of review based on the purpose (Xiao & Watson, 2017)

Purpose Literature Review Methodology Description

review type description
Test Qualitative Ecological Banning 2005 To provide context to research on a
triangulation particular topic. The reported thematic
findings tables are structured as action-
oriented statements; these are provided
for topics of discussion, challenging
factors, and strategies.
Extend Meta- Noblit and Hare 1988 To draw conclusion about behaviour
ethnography To create second order-interpreations
from concept and then extending
those to third order interpretation

meta- Weed 2005 Follow systematic review process

interpretation Identify sub-categories and create
taxonomy
 Types of review based on the purpose (Xiao & Watson, 2017)

Purpose Literature Review Methodology Description

review type description
Extend Qualiative Meta-study Zhao 1991; Paterson & The use of meta-method, meta theory
Canam 2001 analysis and meta-data analysis to
create an overall meta-synthesis on a
particular topic of interest

Mixed Critical Dixon-Wood et al 2006 To create synthesis arguments from

interpretive coding and translating qualitative and
synthesis quantitative research through
integrative grid
Framework Dixon-Wood et al 2011
Critique All literature Critical review Grant & Booth 2009 To examine the reporting and practices
types of meta-analyses in related fields.
What is systematic review and meta-
analysis?
1. Effectiveness reviews
2. Qualitative reviews
3. Cost/economic reviews
4. Prevalence or incidence reviews
5. Diagnostic test accuracy reviews
6. Etiology and risk reviews
7. Textual synthesis reviews
8. Mixed methods reviews
9. Umbrella reviews
10. Scoping reviews
Process of systematic literature review
Process of systematic literature review
1. FORMULATING PROBLEM

¡ guiding and directing the development of the specific inclusion criteria

for the review.
§ Clarity of the review question assists in:
- developing the protocol,
- facilitates effectiveness in the literature search,
- provides a clear structure for the development of the review.
¡ Particular attributes: PICO, PCC, PICo, etc based on the type of review
 Clarifying and linking the purpose and research question

1. IDENTIFY REVIEW
Guides and directs the development of the specific inclusion criteria
QUESTIONS

Clarity in the review question assists in developing the protocol, facilitates effectiveness
in the literature search, and provides a clear structure for the development of the
scoping review report

• Context -- Population -- Concept

Can have major question and sub- • How evidence is to be mapped?
questions
HOW: 1. FORMULATING REVIEW QUESTIONS

Quantitative
Population Qualitative

Intervention Population

Comparator Interest

Outcome Context
HOW: 1. FORMULATING REVIEW QUESTIONS

Population Intervention
The most important characteristics:
¡ Primary intervention of interest Comparator (control
¡ Demographic factors (e.g. age, gender, ethnicity) group)
¡ Socioeconomic factors ¡ - passive (placebo, no treatment, standard
¡ The settings (e.g. hospitals, community, etc) - active (variation of intervention, a drug, a kind of
therapy)
HOW: 1. FORMULATING REVIEW QUESTIONS

Outcomes
Comparator ¡ identify the primary outcome(s) in order to reach a
clinically relevant conclusion
¡ Passive (placebo, no treatment, standard care)
¡ Secondary outcomes may required
¡ Active (variation of intervention, a drug, a kind of
therapy) ¡ Outcomes should be stated neutrally, covering
benefits and adverse effects

¡ Consider how the type and timing of outcome

measurements impacts on outcome measurement
HOW: 1. FORMULATING REVIEW QUESTIONS
2. Protocol development
 2. PROTOCOL AND
REGISTRATION

• Prior registration of a detailed research plan is very

important.
• In order to make the research process transparent,
primary/secondary outcomes and methods are set in
advance, and in the event of changes to the method, other
researchers and readers are informed when, how, and why.
• Registratoin: PROSPERO
(http://www.crd.york.ac.uk/PROSPERO/), and the
registration number is recorded when reporting the study, in
order to share the protocol at the time of planning.
3. Literature search
DATA BASES
3. LITERATURE SEARCH

¡ The review is guided by a requirement to identify all relevant literature regardless of study
design.
¡ The search terms can be redefined, and the process can be undertaken in a more sensitive
searches as familiarity with the literature is increased
¡ The researcher may not wish to place strict limitations on search terms, identification of
relevant studies, or study selection at the outset.
SEARCH STRATEGY

• Using Medical Term: MeSH

• Using Synonym
SEARCH STRATEGY
 STAGES OF A SYSTEMATIC REVIEW
Example : What is the effectiveness of spiritual intervention to
spiritual well-being of patients with cancer
Develop A focused research question
Population
P Intervention Comparator Outcome Study design

Cancer Spiritual Standard care Spiritual well- RCT

interventions being

Yoga Existential well-

being
Mind-body
therapy
Meditation

etc
4. Study selection
HOW: 4.
LITERATURE SEARCH AND STUDY
SELECTION
HOW: 4.
LITERATURE SEARCH AND STUDY
SELECTION
4. STUDY SELECTION

¡ To eliminate studies that did not address central research question

¡ The criteria might be devised post hoc, based on increasing familiarity with the literature, so then
could be applied to all the citations to determine their relevance
¡ Two reviewers applied the inclusion and exclusion criteria to all the citations.
¡ Full texts were obtained for those studies that appeared to represent a ‘best fit’ with the research
question or abstract was unclear.
¡ A deadline should be set (esp. if time is limited) so that any more studies can not be included.

¡ The next stage requires reviewers to read the full articles to make the final decision about whether
they should be chosen for inclusion in the review.
4. STUDY SELECTION

Inclusion and exclusion criteria

¡ Particular attributes: PICO, PCC, PICo, etc based on the type of review
¡ Type of study
¡ Language
¡ Year
 5. Quality
Assessment
QUALITY ASSESSMENT
¡ reduce information overload (eliminate irrelevant or weak studies)

¡ identify papers that are relevant

¡ distinguish evidence from opinion, assumptions, and beliefs

¡ assess study validity (closeness to the truth)

¡ assess study usefulness (clinical applicability)

¡ recognise potential for bias

(PHCRIS 2017)
HOW TO DO QUALITY ASSESSMENT?

¡ Study type—Before beginning the appraisal it is important to classify the

study design/type being used because this will influence the types of
questions to be asked.
HOW TO DO CRITICAL APPRAISAL?
Read the abstract
¡ Are your issues discussed there?
¡ What are the main findings of the research?
¡ Do you want to know more after reading the abstract?

¡ Does it address a related question?

¡ Are there reasons to doubt the findings without reading the whole article?
HOW TO DO CRITICAL APPRAISAL?
Appraising qualitative research
•Sampling method and theoretical saturation
•Triangulation of methods
•Independent analysis
•Reflexivity
•Respondent validation
•Plausible interpretation
•Analysis should be done using explicit, systematic, justified and reproducible methods
HOW TO DO CRITICAL APPRAISAL?
Available appraisal tools
¡ CASP (Critical Appraisal Skills Programme) - systematic reviews, RCTs, diagnostic, economic,
qualitative, case control, cohort study, clinical prediction rule
¡ Joanna Briggs Institute (JBI Critical appraisal tool):
¡ CEBM (Centre for Evidence-Based Medicine) - systematic reviews, diagnosis, prognosis, therapy/RCT
¡ AMSTAR (Assessing the Methodological Quality of Systematic Reviews) - systematic reviews

¡ AACODS checklist - grey literature [PDF 560 KB]

¡ National Institute for Health and Care Excellence (NICE) - quantitative intervention studies,
quantitative studies for correlations and associations, qualitative studies, economic evaluations
¡ GRADE (Grading of Recommendations Assessment, Development and Evaluation) - RCT,
observational; interrupted time series, before-after studies, cohort studies, case control, cross sectional,
case series, case reports, other
6. Charting data/DATA Extraction
6. CHARTING DATA/DATA EXTRACTION

¡ Charting à adopted from charting in qualitative research

technique for synthesizing and interpreting qualitative data by sifting, charting and
sorting material according to key issues and themes
¡ What information should be recorded from the primary studies
¡ It is important to consider how comparisons between different interventions
can be achieved
¡ Producing a short summary or profile of each study may not help readers who
require making important decisions based on the study findings
6. CHARTING DATA/DATA EXTRACTION

¡ Data extraction process

¡ Provides logical and descriptive summary of the results that aligns with the objective and
question/s

¡ Develop draft charting table or form at the protocol stage to record the key information
¡ Can be an iterative process whereby the charting table is continually updated

¡ PILOT TESTING: trial the extraction form on two or three studies to ensure all relevant
results are extracted
¡ Should be transparent and clear in what methods and how the data extracted .
6. CHARTING DATA/DATA EXTRACTION

Arksey & O’Malley 2005; Armstrong et al, 2011:Valaitis et al. 2012

¡ Charting the results can be an iterative

process whereby the charting table is
continually updated.
¡ It is suggested that the review team ¡ One reviewer extracting and another verifying
become familiar with the source results and the data
trial the extraction form on two or three
sources to ensure all relevant results are
extracted.
¡ This pilot step should be done by at least
two members of the review team.
6. CHARTING DATA/DATA EXTRACTION

¡ Author(s)
¡ Year of publication
¡ Origin/country of origin (where the source was published or conducted)
¡ Aims/purpose
¡ Population and sample size within the source of evidence (if applicable)
¡ Methodology / methods
¡ Intervention type, comparator and details of these (e.g. duration of the intervention) (if
applicable). Duration of the intervention (if applicable)
¡ Outcomes and details of these (e.g. how measured) (if applicable)
¡ Key or important findings that relate to the scoping review question/s.
6. CHARTING DATA/DATA EXTRACTION
Author (year) Journal Country Design Population/ Objective Main result
sample
HOW: 6. DATA EXTRACTION

Iskandar et al (2021)
7. Analyze and synthesize
7. Analyze and synthesize

¡ Results may be presented as a map of the data extracted from the included
papers in a diagrammatic or tabular form, and/or in a descriptive format

¡ Elements of the PCC inclusion criteria may be useful to guide how the results should
be mapped
¡ Narrative summary should accompany the tabulated and/or charted results and
should describe how the results relate to the review objective and question/s
HOW: 7. DATA SYNTHESIS

QUANTITATIVE QUALITATIVE
HOW: 7. DATA
SYNTHESIS

¡ STATA

¡ JBI Sumari (quan, qual)

¡ RevMan*

¡ Jamovi*
¡ Rstudio*
HOW: 7. DATA SYNTHESIS

¡ The aim of a meta-analysis is to derive a conclusion with increased power and

accuracy than what could not be able to achieve in individual studies.

¡ It is crucial to evaluate the direction of effect, size of effect, homogeneity of effects

among studies, and strength of evidence.

¡ The data are reviewed qualitatively and quantitatively. If it is determined that the
different research outcomes cannot be combined, all the results and characteristics of
the individual studies are displayed in a table or in a descriptive form; this is referred to
as a qualitative review.
¡ A meta-analysis is a quantitative review, in which the clinical effectiveness is
evaluated by calculating the weighted pooled estimate for the interventions in at least
two separate studies.
 HOW: 7. DATA SYNTHESIS

¡ Heterogeneity among studies à tested using Cochrane’s Q statistic. The degree of heterogeneity was assessed using
the I2 statistic, with I2 values of 25%, 50%, and 75% being considered to indicate low, moderate, and high heterogeneity,
respectively.

¡ In order to analyze effect size, two types of models can be used: a fixed-effect model or a random-effect model.
¡ A fixed-effect model assumes that the effect of treatment is the same, and that variation between results in different
studies is due to random error. Thus, a fixed-effect model can be used when the studies are considered to have the same
design and methodology, or when the variability in results within a study is small, and the variance is thought to be due to
random error.
¡ A random-effect model assumes heterogeneity between the studies being combined, and these models are used when
the studies are assumed different, even if a heterogeneity test does not show a significant result.
HOW: 7. DATA SYNTHESIS
HOW: 7. DATA SYNTHESIS

¡ Heterogeneity among studies à tested using Cochrane’s Q statistic. The degree of heterogeneity was assessed using
the I2 statistic, with I2 values of 25%, 50%, and 75% being considered to indicate low, moderate, and high
heterogeneity, respectively.
¡ Pooled prevalence and 95% confidence intervals (CIs) for outcome à calculated using a random-effects model when
the Cochrane’s Q statistic detected significant heterogeneity; otherwise, a fixed-effects model was used. P < 0.05 was
the threshold for statistical significance..
¡ To assess the risk factors à the odds ratios (ORs) and associated 95% CIs should be extracted from included studies,
and all eligible available data were summarized.

¡ Publication bias à identified by using funnel plot and the asymmetry was tested by using Egger’s linear regression
method (p < 0.1 is considered as significant).
 HOW: 7. DATA
SYNTHESIS (META-
ANALYSIS)
The pooled estimate is the outcome of the meta-
analysis, and is typically explained using a forest plot.
The black squares in the forest plot are the odds ratios
(ORs) and 95% confidence intervals in each study.

The area of the squares represents the weight

reflected in the meta-analysis. The black diamond
represents the OR and 95% confidence interval
calculated across all the included studies. The bold
vertical line represents a lack of therapeutic effect (OR
= 1); if the confidence interval includes OR = 1, it
means no significant difference was found between
the treatment and control groups.
was found between the treatment and control groups.
HOW: 7. DATA SYNTHESIS: QUALITATIVE

¡ A meta synthesis or meta-agregrative is a synthesis for qualitative data

¡ Each finding is rated (unequivocal, credible, or unsupported) to levels of

credibility.
¡ Findings judged to be credible or unequivocal were then integrated into
categories based on similarity of meaning.
¡ Findings categorized as unsupported were not included in the subsequent
analysis.
¡ The final step of the data analysis was conducted by grouping the categories
into synthesized findings.
HOW: 7. DATA SYNTHESIS: QUALITATIVE
7. Reporting

¡ PRISMA (https://www.prisma-statement.org/)

¡ PRISMA checklist
¡ PRISMA flow diagram
REFERENCES
¡ Bialocerkwoski, A., Klup, N., Bragge, P (2010). How to read and critically appraise a reliability article.
International Journal of Therapy and Rehabilitation, vol 7 (3), pp. 114-120).

¡ Burls A. What is critical appraisal? 2009

¡ Hilary Arksey & Lisa O'Malley (2005) Scoping studies: towards a methodological framework, International Journal of
Social Research Methodology, 8:1, 19-32, DOI: 10.1080/1364557032000119616

¡ Primary Health Care Research & Information Service (2017). PHCRIS Getting Started Guides: Introduction
to... Critical appraisal of literature. From http://www.phcris.org.au/guides/critical_appraisal.php (Accessed 4
Nov 2017)

¡ Xiao, Y, Watson M (2017). Guidance on Conducting a Systematic Literature Review