Questionnaire Development

and Testing

Barbara J. Stussman
Survey Statistician
NIH/NCCIH
 Acknowledgments

• Gordon Willis, Ph.D.

Applied Research Program, NCI

• William T. Riley, Ph.D.

Director
Office of Behavioral and Social Sciences
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative Research
• Mixed Methods Research
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative Research
• Mixed Methods Research
 Data Sources for Health
Research (other than self-report)

 Birth and death records

 Medical records at physician offices,
hospitals, nursing homes, etc.
 Medical databases housed within
various agencies, universities, and
insurance companies
 Physical exams and laboratory testing
 Registers of diseases
 Self-Report Measures
• Used in questionnaires

• Some information can be gathered only

by asking people questions (i.e. not easily
observable)

• Self-report measures are estimates of true

scores
True score + Measurement error =
Survey response
 What are the Pitfalls
of Self-Report?

Susceptible to the respondent’s:

Mood
Motivation
Memory
Understanding
 What are the Pitfalls
of Self-Report?

Also susceptible to:

 Context of interview

 Social desirability

Thus, importance of rigorous methods

Common Types of Questions
• Open-ended
 What health conditions do you have?

• Closed
 Which of the following conditions do
you currently have? Say yes or no to
each.

 Diabetes?
 Asthma?
 Hypertension?
Common Types of Questions
• Response options
 Nominal – unordered (e.g. male, female)
 Ordinal – ranked (e.g. excellent, good,
fair, poor)

• Type of information
 Factual – Objectively verifiable
 Subjective – Knowledge, perceptions,
feelings, judgment
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative Research
• Mixed Methods Research
 Data Collection Methods in
Surveys
• Computerized vs. paper surveys

 Computerized: desktop, laptop, web,

smartphone, tablet

• Interviewer vs. self-administered

 Interviewer: Face-to-face or from

centralized location
 Occasionally interactive voice response
Skip Patterns Have you ever had
asthma?

YES NO END

Have you had an asthma

attack in the past 12 months?

YES NO END

How many times have you gone to the

emergency room because of asthma?
 Computerized Surveys

Pros

 Faster data availability

 Can handle complex skip patterns
 Can help to eliminate skip errors (but
not always)
 Can be tailored to severity of
symptoms or situation
 Computerized Surveys

Cons

 Data can get lost if system

crashes
 Requires power source
In-Person Interviewer Administered
Pros
 Interviewer can answer questions
 Can administer to illiterate/low
reading level
 Can reach people who can’t come to
you
 Can build rapport
 Higher response rates
 Can use visual aids
 In-Person Interviewer
Administered
Cons

 Expensive
 Geographic limitations
 Longer data collection period
 Interviewer presence can bias results
 Interviewers may use inconsistent
techniques
Telephone Interviewer Administered

Pros

 Lower Costs
 Can ensure uniform data collection
 Shorter data collection period
 Good geographic coverage
Telephone Interviewer Administered

Cons

 Omit people without phones (2% of US)

 4 in 10 U.S. adults are cell phone only
(complicates sampling)
 Cannot use visual aids
 Lower response rates compared with
in-person
 Web/Smartphone/Tablet
Self-Administration
Pros

 Lower costs
 Timely data
 Anonymity (good for sensitive items)
 Flexible in design options (can use visual
aids, long lists, complex skips)
 Convenient for respondents (any
time/location)
 Can cover large geographic area
 Web/Smartphone/Tablet
Self-Administration
Cons

 Varying degrees of computer skills,

access, connection speeds
 Samples reflect select online groups
 Difficult to verify informed consent
 Difficult to track non-responders
 Paper and Pen Self-Administration
Pros

 Anonymity for sensitive questions

 Can use long, complex response
categories
 Can use visual aids
 Appearance consistent
 Can cover large geographic area
 Length easy to see (plus or minus?)
 Paper and Pen Self-Administration

Cons

 Requires good reading/writing skills

 Cannot have complex skip patterns
 No quality control
 Does not always save money
 Effects of Data Collection
Method on Response
• Multiple methods increase response rates
(but at what cost?)
 Spoken vs. Visual
 Spoken questions produce more
positive responses

“How would you describe your health,

would you say excellent, good, fair or
poor?”
 Effects of Data Collection
Method on Response
• Questions tailored to method:
 Yes/No for telephone
 Long list of check boxes for web
 Long scales for self-administered/
shorter scales for telephone
 Vast array of visuals/graphics available
for computerized surveys

 Be careful combining/comparing
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Main Development Steps:
1. Determine Analytic Objectives

 What are the general concepts to be

covered/research questions?

 Literature review
 Expert panels, think tanks
 Patient input
 1. Determine Analytic Objectives
What Type of Data Will Answer the
Research Question?

% of % who took a % who had

respondents yoga class in insurance
who used yoga past 12 months coverage for
in past 12 cost of yoga
months class

Men (X%) (X%)

(X%)
Women (X%) (X%)
(X%)
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Main Development Steps:
2. Put Together Draft Questionnaire
Use existing instruments
 National Field Surveys:
 National Health Interview Survey (US)
 National Health And Nutrition Examination
Survey (US)
 National Health Service Survey (UK)
 Research Electronic Data Capture
(REDCap):
 Shared library of data collection
instruments
Put Together Draft Questionnaire

HealthMeasures.net (sponsored by NIH/

developed with best practices)
 Patient Reported Outcomes Measurement
Information System (PROMIS)
 Quality of Life in Neurological Disorders
(Neuro-QoL)
 Adult Sickle Cell Quality of Life
Measurement Information System (ASCQ-
Me)
 NIH Toolbox
 Put Together Draft Questionnaire

• Pay attention to aesthetics

• Draft new questions using known

criteria
Put Together Draft Questionnaire
• Literacy < 9th grade U.S.
• Specific better than broad
• Culturally sensitive
• Scales consistent
• Terms well-defined
• Instructions clear
• Reference periods clear
• Response options match question
• Multiple concepts separated
Put Together Draft Questionnaire
 Interpreted accurately by people with
range of demographic characteristics
 Capturing what researcher intended
Avoid
 Social desirability effects
 Negative wording
 Double barreled
 Jargon
 Ambiguous
 Leading
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Main Development Steps:
3. Cognitive Testing

 Cognitive Testing

Technique to study the way in which

respondents understand, process, and
respond to survey questions

 Probing techniques to determine how

respondents interpret questions
 Cognitive Testing
 All components tested (stem, response
categories, instructions, question ordering)

 Qualitative analysis performed to find

common themes

 Performed in laboratory by trained research

team

 Can be beneficial to travel to respondents

(hard to reach populations)
 Cognitive Testing

Participants:

• Based on target population

• Start to lose focus after 1 hour

• Usually paid
 Cognitive Testing
Iterative Process

Test questionnaire

Modify based on
cognitive testing

• Goal: Find and fix sources of Response Error

Sources of Response Error
 Respondent does not know the
information
 They cannot recall it, although they do
know it
 They do not understand the question
 They do not want to report the answer in
the survey context

(Fowler F. (1991). Survey Research

Methods, Sage)
 Actions Taken Based on Cognitive
Interviews

• Accept original question

• Accept original question with minor edits

• Accept original question with major edits

• Drop question/draft new question

Examples of common questionnaire
problems: Too Broad

Original

Would you be more inclined to use

complementary therapies if your medical
doctor or other conventional health care
provider recommends it?
Examples of common questionnaire
problems: Too Broad

Final

During the past 12 months, did you use

[therapy] because it was recommended by a
medical doctor?
Examples of common questionnaire
problems: Too Complex
Original
During the past 30 days did you use any
of the following vitamins and minerals for
your own health or treatment? Be sure to
include ALL vitamins that you use. If you
take a SINGLE vitamin or mineral
supplement, such as niacin, that is not
part of a combination multi-
vitamin/mineral supplement, include it
separately.
Examples of common questionnaire
problems: Too Complex

Final

The next questions are about any

vitamins and minerals you may take.
Have you ever taken any vitamins or
minerals listed on this card?
Examples of common questionnaire
problems: Double-barreled

Original

What was the reason you chose to

use acupuncture…was it to treat a
specific health problem or just to
stay healthy or well?
Examples of common questionnaire
problems: Double-barreled

Final

Did you use acupuncture for any of these

reasons? Please say yes or no to each.

For general wellness or general

disease prevention?

For one or more specific health

problems, symptoms, or conditions?
Examples of common questionnaire
problems: Information Unknown
Questions dropped

• Do you currently see a practitioner for

homeopathy more, less, or about the
same as you did one year ago?

• At what age did you first start using

[complementary therapy]?

• During the past 12 months, did your child

pray for his/her own health?
Examples of common questionnaire
problems: Terms Undefined

Original

During the past 12 months did you use

movement therapies for your own health
or treatment?
Examples of common questionnaire
problems: Terms Undefined
Final

Have you ever practiced any of the

following movement or exercise
techniques?

 Alexander Technique?
 Feldenkrais?
 Pilates?
 Trager Psychophysical Integration?
Examples of common questionnaire
problems: Cultural salience
 Have you ever switched from a stronger to
a lighter cigarette?

 (Original) During the past 12 months, did

you see a practitioner for/use [therapy]
because it is how you were raised?

 (Final) During the past 12 months, did you

see a practitioner for/use [therapy]
because it was part of your upbringing?
 Examples of common questionnaire
problems: Inconsistent response categories

• NHIS: 10-fold increase in children

reported to have cerebral palsy 2004-
2006
 311-353 cases in 2004-2006 vs. 34
cases in 2003

• Caused by interviewer error when

survey moved to new screen design
Looking at this list, has a doctor or other health
professional ever told you that {child name}
had any of these conditions?

(00) None
(01) Down’s syndrome
(02) Cerebral Palsy
(03) Muscular Dystrophy
(04) Cystic Fibrosis
(05) Sickle cell anemia
(06) Autism
(07) Diabetes
(08) Arthritis
(09) Congenital heart disease
(10) Other heart condition
 Examples of common
questionnaire problems:
Ordering effects

Original
Have you ever had a sigmoidoscopy?
When was your most recent?

Have you ever had a colonoscopy?

When was your most recent?

Final
Order reversed
 Pay Attention to Length
• Ideally
 < 30 minutes for face-to-face
 <15 minutes for phone or web

• Too long will increase costs/decrease

response rates
 Interviewers rush
 Respondents get tired
 Interviewers may cheat (keystroke data)
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Main Development Steps:
4. Field Pretesting
• Survey administered in realistic setting
to similar study population

• For Interviewer-administered surveys

 Experienced interviewers
 Nearly final instrument
 Designers/sponsors observe
 Rating forms to record issues
 Debriefing
 Field Pretesting
• For Self-administered surveys

 Respondents interviewed after

they complete survey

 Observe respondents as they fill

out survey
 Field Pretesting
• Tabulated data used to:

 Design closed response categories

from open-ended questions
 Collapse/eliminate response
categories
 Alter skip patterns
 Drop items
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 5. Translation
PROMIS Approach

 Harmonization: different words/

languages must mean the same

 Universal approach: One language

version for multiple countries
 People from various
countries/dialects involved
 Translation (PROMIS approach)

1. 2 English to target lang. translations

2. Native speaker of target lang. reconciles

3. Back translated by native English speaker

4. Review by project manager

5. 3 native target lang. experts review (linguists

and healthcare professionals)

6. Review by project manager

 Translation (PROMIS approach)
7. Native target lang. speaker reviews history of
items/determines final version

8. Review by project manager

9. Formatting/typesetting/proofreading

10. Cognitive testing with native target lang.

speakers

11. Compilation of comments and finalization

 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Qualitative Research

The systematic collection,

organization, and interpretation
of textual material derived from
talk or observation.

(Malterud, 2001)
 Qualitative Research
Quantitative Qualitative

• Begin w/ hypotheses  Generate hypotheses

• Specific research  General research

questions questions

• Analysis after data  Analysis ongoing

collection during data collection
 Qualitative Research
Data collection:

• Data sources: interviews, observations, videos,

diaries, memoirs, biographies

• Requires sensitivity to pick up on non-verbal

cues
 Familiarity with literature can enhance
sensitivity

• Context is important
 Qualitative Research
Data collection:

• Interviews audiotaped/transcribed later

• Interview protocol
 Data about date, time, location
 Questions to be asked during interview

 Additional questions asked

spontaneously
 Qualitative Research
Data Analysis:

• Begins with 1st piece of data

• Coding: assigning themes to data
• Each piece of data compared for
similarities/differences
• Conceptually similar segments
grouped to form categories
• Concepts form basis of analysis
 Qualitative Research
Data Analysis:

• Coding scheme evolves

• Saturation: no new piece of data challenges
categorical structure
• Both an art and a science
• More than one story can be derived from
data
• Qualitative data analysis software (e.g.
MAXQDA, ATLAS, Nvivo)
 Outline of Lecture
• Self-report measures
• Data collection methods
• Main questionnaire development steps
1. Determine analytic objectives
2. Put together draft questionnaire
3. Cognitive testing
4. Field pretesting
5. Translation
• Qualitative research
• Mixed Methods Research
 Mixed Methods Research
Integrating or combining qualitative and
quantitative methods to draw on strengths of
each

Reasons for using

 View problems from multiple perspectives
 Contextualize information
 Develop more complete understanding
Challenges
 Teamwork, resources, sample size,
interpretation
 Mixed Methods Designs
 Qualitative → Quantitative: Qualitative
research used to develop outcome
measures or intervention

 Most pertinent to questionnaire

development

 Qualitative data used to ensure

instrument is grounded in views of
participants
 Mixed Methods Designs
Example:

• Patient Reported Outcomes Measurement

Information System (PROMIS):

Expert review → focus groups →

cognitive testing → administer survey →
psychometrics
 Mixed Methods Designs
Quantitative → Qualitative: Qualitative used to
help explain the quantitative data

Example: Do positive views on aging influence

health?
 First questionnaires used to determine if
there are associations
 Then qualitative interviews used to determine
specific barriers and resources that impact
health behaviors

Craciun et al. (2015)

 Mixed Methods Designs
Qualitative and Quantitative used Concurrently:
Both methods used at the same time to answer the
same research question

Example: Sought to better understand cause of

distracted driving by commercial truck drivers
• Qualitative interviews identified how supervisors
might distract drivers
• Surveys focused on decision-making by drivers
in near crashes
• Synthesis of these guided interventions

Swedler et al. (2015)

 Mixed Methods Research
NIH Clinical Center Example

 Brain imaging study with Fibromyalgia

patients and healthy volunteers

 Subjects given experimental heat on leg

using thermode, and asked to rate their
level of pain

 Qualitative component added to learn how

patient’s determine pain ratings (looking
for differences between FM and HV)
 Mixed Methods Research
NIH Clinical Center Example

 Qualitative data used to explain peculiarities

in quantitative data:
 Identified subjects who were confused/did
not follow directions
 Identify subjects who had trouble
focusing/fell asleep
 Identify subjects who had pain other than
FM
 Summary

• Questionnaire development requires

careful planning

• Use existing validated instruments when

possible

• Rigorous methods will reduce response

error
Suggested Resources (Textbooks)
• Cognitive Interviewing (Willis, 2005/Sage)

• Survey Research Methods (Fowler,

2014/Sage)

• Basics of Qualitative Research 3e (Corbin

and Strauss, Sage/2008)

• Designing and Conducting Mixed Methods

Research (Creswell and Clark, 2007/Wiley)

• Mail and Web Surveys (Dillman, 2007/Wiley)

 Suggested Resources
(Internet)
• Question Appraisal System (Willis & Lessler,
1999)
 (appliedresearch.cancer.gov/areas/
cognitive/qas99.pdf)

• NIH sponsored health measures

 (healthmeasures.net)

• Patient Reported Outcomes Measurement

Information System (PROMIS)
 (nihpromis.org)
 Suggested Resources
(Internet)
• NIH Toolbox
 (nihtoolbox.org)

• Quality of Life in Neurological Disorders

 (neuroqol.org)

• Adult Sickle Cell Quality of Life Measurement

Information System
 (ascq-me.org)
 Suggested Resources
(Internet)
• United Health Service Surveys (U.K.)
(nhssurveys.org/)

• Centers for Disease Control and Prevention

(CDC) Surveys
 (cdc.gov/nchs/surveys.htm)

• REDCap Shared Library

 (project-redcap.org)