Survey Research Methods

A Fink, University of California, Los Angeles, Los Angeles, CA, USA; The Langley Research Institute, Pacific
Palisades, CA, USA
ã 2010 Elsevier Ltd. All rights reserved.

Glossary individuals’ obtained scores are different from their

true scores, which can only be obtained from perfect
Categorical or nominal – Categorical or nominal measures.
response choices have no numerical or preferential Survey – A system for collecting valid information
values. from or about people to describe, compare, or
Concurrent validity – Demonstrated when two explain their knowledge, attitudes, and behavior
assessments agree or a new measure is compared (Fink, 2008). The system consists of interrelated
favorably with one that is already considered valid. activities starting with defining precise survey
Construct validity – Established experimentally to objectives, choosing respondents, preparing a
demonstrate that a survey distinguishes between reliable and valid survey instrument, testing the
people who do and do not have certain survey with respondents, and conducting all activities
characteristics. It is usually established in an ethical manner.
experimentally. Test–retest reliability or stability – A measure of a
Content validity – Refers to the extent to which a survey’s stability over time. A survey is stable if the
measure thoroughly and appropriately assesses the correlation between scores from one time to another
skills or characteristics it is intended to measure. is high.
Face validity – Refers to how a measure appears on
the surface: Does it seem to ask all the needed
questions?
Survey’s Internal Consistency or homogeneity –
A survey’s internal consistency or homogeneity
Refers to the extent to which all the items or Survey Research: Definitions and Uses
questions assess the same skill, characteristic, or
quality. A survey is a system for collecting valid information from
Nonprobability or convenience samples – They or about people to describe, compare, or explain their
are chosen based on judgment regarding the knowledge, attitudes, and behavior (Fink, 2008a). The sys-
characteristics of the target population and the needs tem consists of interrelated activities starting with defining
of the survey. With nonprobability sampling, some precise survey objectives, choosing respondents, preparing
members of the eligible target population have a a reliable and valid survey instrument, testing the survey
chance of being chosen and others do not. with respondents, and conducting all activities in an ethical
Numerical – Response choices call for numbers manner. The survey researcher is responsible for imple-
such as age (e.g., number of years) or height (e.g., menting all survey activities and ensuring their quality.
number of meters). Survey information can be collected directly by asking
Ordinal – Response choices require rate or order people to answer questions, or indirectly, by reviewing
choices, say, from very positive to very negative. written, oral, and visual records of people’s thoughts and
Predictive validity – Refers to the extent to which a actions. Survey data can also be obtained by observing
survey measure forecasts future performance. people in natural or experimental settings.
Probability sampling – It provides a statistical basis Surveys are taken of political and consumer choices,
for saying that a sample is representative of the study use of health services, numbers of people in the labor
or target population. In probability sampling, every force, and opinions on just about everything from aard-
member of the target population has a known, varks to zyzyvas. Individuals, communities, schools, busi-
nonzero probability of being included in the sample.’ nesses, and researchers use surveys to find out about
Probability sampling implies the use of random people by asking questions about feelings, motivations,
selection. plans, beliefs, and personal backgrounds. The questions
Reliable survey instrument – A reliable survey in survey instruments are typically arranged into mailed
instrument is one that is relatively free from or self-administered questionnaires (on paper or online)
‘‘measurement error.’’ Because of this ‘‘error,’’ and in-person (face-to-face) or telephone interviews.

152
 Survey Research Methods 153

Surveys are a prominent part of life in many major smoking in students, and a component of that study may
industrialized nations, particularly the United States. US be a survey of students’ smoking habits.
elections are always accompanied by a flood of polls. The A survey’s objectives can be derived from reviews of
US Census, which is administered to the entire nation the literature and other surveys. The literature refers to
every 10 years, is a survey. However, most people encoun- all published and unpublished public reports on a topic.
ter surveys more frequently in their daily lives in medical, Systematic reviews of the literature describe current
educational, and social settings. knowledge and reveal gaps. For instance, a review of the
literature can provide information on best practices in
teaching reading but may not provide sufficient informa-
Survey Objectives tion about the ease of implementing such programs in
classrooms. A survey of reading teachers may provide data
A survey’s objectives are general statements of the sur-
on implementation.
vey’s outcomes and provide the direction for selecting
Survey objectives can also come from experts. Experts
questions (Case study 1).
are individuals who are knowledgeable about a topic, will
When planning a survey, the survey researcher must
be affected by the survey’s outcomes, or are influential in
define all potentially imprecise or ambiguous terms in the
implementing its findings. Experts can be asked about
objectives (Fink, 2004a). For the objectives above, the
objectives by mail or telephone or brought together in
imprecise terms are needs; educational services; charac-
meetings. Two types of meetings that are sometimes used
teristics; and benefits. No standard definition exists for
to help survey researchers identify objectives, research ques-
any of them. What are needs, for example, and of the very
tions, and research hypotheses are focus groups (Krueger
long list that the survey researcher can create, which are
and Casey, 2000) and consensus panels ( Jones, 1995).
so important that they should be included on the survey?
Definitions can come from the literature and from con-
sultation with knowledgeable individuals.
Straightforward Questions and
Survey objectives may be independent of an existing
Responses
study or related to it. For instance, suppose a school
district decides to investigate the causes of a measurable
Survey questions take two primary forms. When they
increase in smoking among students between 12 and
require respondents to use their own words, they are
16 years of age. The district could then ask the survey
called open ended. When the answers or responses are
research department to design and implement a survey of
preselected for the respondent, the question is termed
students who smoke to find out why they do. On the other
closed or forced choice. Both types of questions have
hand, the school district may be part of a study to prevent
advantages and limitations.
An open question is useful when the intricacies of
an issue are still unknown, for eliciting unanticipated
Case study 1 answers, and for describing the world as the respondent
Illustrative objectives for a survey of vocational educational sees it – rather than as the questioner does. Some respon-
needs dents also prefer to state their views in their own words.

Objective 1: Identify the most common needs for educational Sometimes, when left to their own devices, respondents
services for three professions: teaching, nursing, and com- provide quotable material. The disadvantage is that unless
puter programming. you are a trained anthropologist or qualitative researcher,
Sample question:
How proficient would you say you are in performing the
responses to open questions are often difficult to compare
following job-related activities? and interpret.

Objective 2: Compare the educational needs of men and Some respondents prefer closed questions because
women they are either unwilling or unable to express themselves
Question: while being surveyed. Closed questions are more difficult
Are you male or female?

Objective 3: After participation in a vocational education pro-
to write than open ones, however, because the answers or
gram, identify the characteristics of participants who receive response choices must be known in advance; however, the
the most benefits. results lend themselves more readily to statistical analysis
Questions about the characteristics of survey participants: and interpretation, and this is particularly important in
What is your occupation? What was your household income large surveys because of the number of responses and
last year?
Questions about benefits:
respondents. Moreover, as the respondent’s expectations
To what extent has this program helped you improve your job are more clearly spelled out in closed questions (or the
skills? How long did you wait to get a job in your preferred survey researcher’s interpretations of them), the answers
profession? have a better chance of being more reliable or consistent
over time.
154 Education Research Methodology: Quantitative Methods and Research

Survey Question Stems Progress. For demographic questions, researchers often

search the literature using common article databases of
Open-ended survey questions have only a stem: Why
Education Resources Information Center (ERIC), US
were you dissatisified with the workshop? Close-ended
Census Bureau, or the Web of Science.
questions have a stem and one or more responses: Which
of the following describes the reasons for your dissatisfac-
Responses
tion? (The question is followed by a list of potential reasons
for dissatisfaction.) The choices given to respondents for their answers can
A straightforward survey question asks what it needs in take three forms. The first is called nominal or categorical.
an unambiguous, concrete way and extracts accurate and (The two terms are often used interchangeably.) Categor-
consistent information (Sudman and Bradburn, 1982; ical or nominal choices have no numerical or preferential
Fink, 2004b). Straightforward questions are purposeful values. For example, asking respondents if they are male
and use correct grammar (Case study 2). or female asks them to name themselves as belonging to
Question stems should contain fewer than 20 words one of two categories: male or female. Any number asso-
and be easy to understand, nonjudgmental, and unbiased. ciated with either category (e.g., 1 ¼ male) has no inherent
They should be phrased in a socially and culturally sensi- importance. The second response form is called ordinal.
tive manner and at a reading level that is appropriate for When respondents are asked to rate or order choices, say,
the respondents. The survey researcher should avoid from very positive to very negative, they are given ordinal
abbreviations and absolute terms (never, always). Ques- choices. The third response form, numerical, asks for num-
tion writing is complex, and requires ability to write bers such as age (e.g., number of years) or height (e.g.,
simply. For many topics (e.g., perceptions, values, feelings, number of meters). Numerical responses produce data
and beliefs) question writing also requires the researcher that are continuous (e.g., age) or discrete (e.g., number of
to understand psychological, social, and health theories books read in the past month). Categorical, ordinal, and
and research. The difficulty in question-writing makes numerical response choices are illustrated in Case study 3.
survey researchers often adopt or adapt already-existing
and commonly used questions and even entire survey
Survey Instruments
instruments to avoid the painstaking and often painful
process of writing questions on their own. The Web Self-Administered Questionnaires
has numerous sources for questions. For instance, ques-
A self-administered questionnaire consists of questions
tions on reading and math in schools can be found on
that an individual completes by oneself. These question-
the website of the National Assessment of Educational
naires may be completed on site (in a classroom or clinic)
or at a distant location (home). Respondents receive self-
Case study 2 administered questionnaires in the mail or are directed
(via email) to a website. Many survey software vendors
Survey questions
provide easily accessible and relatively inexpensive pro-

Purposeful questions: The relationship between the question grams that enable the novice to design easy-to-use
and the survey’s objective surveys that analyze survey responses in real time and

Survey objective: To find out about reading habits in the
provide user-friendly reports.
community

Survey question: In what year were you born?

Comment: The relationship between the question and the Interviews
objective is far from clear. An explanation is needed.
An interview requires at least two people: one to ask the
Complete sentences: questions (the interviewer) and another to respond (the

Poor: Place of birth? interviewee). (Group interviews are possible.) Interviews

Comment: Place of birth means different things to different can take place on the telephone, face to face, or via
people. I might give the city in which I was born, but you might teleconference or web cam. Telephone interviews may
tell the name of the hospital.
be relatively simple, involving just an interviewer and

Better: Name the city and state in which you were born.
interviewee, or they may be relatively complex, relying
Concrete questions: on advanced technology to identify and query respon-

Less concrete: How would you describe your mood? dents (Bourque and Fielder, 2004).

More concrete: In the past 3 weeks, would you say you were
generally happy or generally unhappy? Structured Record Reviews

Comment: Adding a time period (3 weeks) and defining mood
(generally happy and generally unhappy) adds precision to the A structured record review is a survey that uses a specially
question. created questionnaire to guide the collection of data from
financial, medical, school, and other records including
 Survey Research Methods 155

For example, two visitors to school are participants in a

Case study 3
structured observation if both are asked to count and
Categorical, ordinal, and numerical responses record the number of computers they see, look for the
1. Nominal response: Which of these books have you read? presence or absence of air conditioning, and measure the
Please circle yes or no for each choice. room’s area in square feet.

Book title 1. Yes 2. No

Survey Sampling
Tom Sawyer 1 2
Tale of Two Cities 1 2
Etc. (six books are listed)
A sample is a portion or subset of a larger group called
a population. Surveys often use samples rather than
2. Ordinal: How important has each of the following books been
populations. A good sample is a miniature version of the
in helping form your image of modern life? Please use the population – just like it, only smaller. The best sample is
following scale to make your rating: representative, or a model of the population. A sample is
1 = Definitely unimportant representative of the population if important characteris-
2 = Probably unimportant tics (e.g., age, gender, and reading level) are distributed
3 = Probably important
4 = Definitely important
similarly in both groups. Suppose the population of inter-
5 = No opinion/Don’t know est consists of 1000 people, 50% of whom are male, with
45% over 13 years of age and older. A representative
sample will have fewer people, say, 500, but it must
Book title Please circle one for each also consist of 50% males, with 45% over age 13 years
book and older.
Tom Sawyer 1 2 3 4 5
No sample is perfect. Usually, it has some degree of
Tale of Two Cities 1 2 3 4 5 bias or error. To ensure a sample whose characteristics
Etc. (six books are listed) and degree of representation can be described accurately,
the survey researcher must start with very specific and
3. Numerical: How many books did you read in the past month? precise survey objectives. The researcher also must have
_____________ clear and definite eligibility criteria, apply sampling meth-
The response choice format dictates the analysis and report ods rigorously, justify the sample size, and have an ade-
of results. For example, a report of the responses to the first quate response rate.
question yielding categorical data could take this form:
Nominal or categorical data: More than 75% of the
respondents read at least one book on the list, but no one Eligibility Criteria
read all six. Of 75 respondents, 46 (61.3%) indicated they had
read Tom Sawyer, the most frequently read book. The criteria for inclusion in a survey refer to the charac-
teristics of respondents who are eligible for participation
A report for the second question (ordinal data) might look
like this:
in the survey; the exclusion criteria consist of characteris-
tics that rule out certain people. The survey researcher
Ordinal data: Of 75 respondents completing this question,
applies the inclusion and exclusion criteria to the target
43 (57.3%) rated each book as definitely or probably important.
The average ratings ranged from 3.7 for Tom Sawyer to 2.0 for population. Once the researcher removes from the target
A Tale of Two Cities. population all those who fail to meet the inclusion criteria
Typical survey findings for the third question (numerical)
and all those who succeed in meeting the exclusion criteria,
might appear as follows: the remaining study population, is eligible for participation.
A major reason for setting eligibility criteria is that to do
On average, respondents read about one book each month.
otherwise is simply not practical. Setting inclusion and
exclusion criteria is an efficient way of focusing the survey
electronic, written, and filmed documents. An example of a on only those people from whom you are equipped to get
structured record review is the use of a questionnaire to the most accurate information or about whom you want to
collect information from school attendance records on the learn something.
number and characteristics (e.g., age, reading level) of
students who are absent 4 or more weeks each semester.
Sampling Methods
Structured Observations
Sampling methods are often divided into two types.
A structured observation is designed to guide the ob- The first is called probability sampling. Probability sam-
server in focusing on specific actions or characteristics. pling provides a statistical basis for saying that a sample is
156 Education Research Methodology: Quantitative Methods and Research

representative of the study or target population, and every are related to the outcome – in this case, care for health
member of the target population has a known, nonzero needs of homeless families. The justification for the selec-
probability of being included in the sample. Probability tion of the strata can come from the literature and expert
sampling implies the use of random selection. Random opinion.
sampling eliminates subjectivity in choosing a sample. Stratified random sampling is more complicated than
It is a fair way of getting a sample. simple random sampling. The strata must be identified
The second type of sampling is nonprobability or conve- and justified, and using many subgroups can lead to large,
nience sampling. Nonprobability samples are chosen based unwieldy, and expensive surveys.
on judgment regarding the characteristics of the target Systematic sampling is another sampling method used
population and the needs of the survey. With nonprobability by survey researchers. Suppose a researcher has a list with
sampling, some members of the eligible target population the names of 3000 students, from which a sample of 500 is
have a chance of being chosen and others do not. By chance, to be selected. Dividing 3000 by 500 yields 6, which means
the survey’s findings may not be applicable to the target that one of every six persons will be in the sample. To
group at all. True probability sampling is extraordinarily systematically sample from the list, a random start is
complex. Nevertheless, many survey researchers aim to needed. To obtain this, a die can be tossed. Suppose a
come as close to a probability sample as possible. One toss comes up with the number 5. This means the fifth
approach is simple random sampling. name on the list would be selected first, then the 11th,
In simple random sampling, every subject or unit has 17th, 23rd, and so on until 500 names are selected.
an equal chance of being selected. Members of the target To obtain a valid sample, the researcher must obtain
population are selected one at a time and independently. a list of all eligible participants or members of the popu-
Once they have been selected, they are not eligible for a lation. This is called the sampling frame. Systematic
second chance and are not returned to the pool. Because sampling should not be used if repetition is a natural
of this equality of opportunity, random samples are con- component of the sampling frame. For example, if the
sidered relatively unbiased. Typical ways of selecting a frame is a list of names, systematic sampling can result
simple random sample are using a table of random num- in the loss of names that appear infrequently (e.g., names
bers or a computer-generated list of random numbers and beginning with X). If the data are arranged by months and
applying it to lists of prospective participants. the interval is 12, the same months will be selected for
The advantage of simple random sampling is that the each year. Infrequently appearing names and ordered data
survey researcher can get an unbiased sample without ( January is always month 1 and December month 12)
much technical difficulty. Unfortunately, random sampling prevents each sampling unit (names or months) from
may not pick up all the elements of interest in a population. having an equal chance of selection. If systematic sam-
Suppose a researcher is conducting a survey of teacher pling is used without the guarantee that all units have an
satisfaction. Consider also that the researcher has evidence equal chance of selection, the resultant sample will not be
from a previous study that older and younger teachers a probability sample. When the sampling frame has no
usually differ substantially in their satisfaction. If the inherently recurring order, or you can reorder the list or
researcher chooses a simple random sample for the new adjust the sampling intervals, systematic sampling resem-
survey, he or she might not pick up a large-enough pro- bles simple random sampling.
portion of younger teachers to detect any differences that A cluster is a naturally occurring unit (e.g., a school,
matter. In fact, with really bad luck, by chance, the entire which has many classrooms, students, and teachers). Other
sample may consist of older teachers. To be sure that the clusters are universities, hospitals, cities, states, and so on.
sample consists of adequate proportions of people with The clusters are randomly selected (called cluster sam-
certain characteristics, the researcher can use stratified pling), and all members of the selected cluster are included
random sampling. in the sample. For example, suppose that California’s
A stratified random sample is one in which the popula- counties are trying out a new program to improve physical
tion is divided into subgroups or strata, and a random education for teens. A researcher who wanted to use cluster
sample is then selected from each subgroup. For example, sampling can consider each county as a cluster and select
suppose a survey researcher wants to determine the effec- and assign counties at random to the new physical educa-
tiveness of a program to care for the health of homeless tion program or to the traditional one. The programs in the
families. The researcher plans to survey a sample of 1800 of selected counties would then be the focus of the survey.
the 3000 family members who have participated in the Cluster sampling is used in large surveys. It differs
program. The researcher also intends to divide the family from stratified sampling in that with cluster sampling
members into groups according to their general health you start with a naturally occurring constituency. The
status (as indicated by scores on a 32-item test) and age. researcher then selects from among the clusters and either
Health status and age are the strata. The strata or sub- surveys all members of the selection or randomly selects
groups are chosen because evidence is available that they from among them. The resulting sample may not be
 Survey Research Methods 157

representative of areas not covered by the cluster, nor the number of eligible respondents (denominator). Prac-
does one cluster necessarily represent another. tically all surveys are accompanied by a loss of informa-
Nonprobability samples do not guarantee that all eli- tion because of nonresponse. These nonresponses may
gible units have an equal chance of being included in introduce error into the survey’s results because of differ-
a sample. Their main advantage is that they are rela- ences between respondents and nonrespondents.
tively convenient, economical, and appropriate for many
surveys. Their main disadvantage is that they are vulner-
able to selection biases. Convenience sampling is a type of Reliable and Valid Survey Instruments
nonprobability sampling in which the researcher surveys
individuals who are ready and available. For example, a A reliable survey instrument is consistent; a valid one is
survey that relies on people in a shopping mall is using a correct (Litwin, 2004) For example, an instrument is reli-
convenience sample. able if each time you use it (and assuming no intervention),
Snowball sampling relies on previously identified you get the same information. But it may not be correct!
members of a group to identify other members of the Reliability or the consistency of information can be seri-
population. As newly identified members name others, ously imperiled by poorly worded and imprecise questions
the sample snowballs. This technique is used when a and directions. If an instrument is unreliable, it is also
population listing is unavailable and cannot be compiled. invalid because inconsistent data are incorrect. Valid survey
For example, surveys of teenage gang members and instruments serve the purpose they were intended to and
undocumented workers might be asked to participate provide correct information. For example, if a survey’s
in snowball sampling because no membership list is objective is to find out about mental health, the results
available. should be consistent with other measures of mental health
Quota sampling divides the population being studied and inconsistent with measures of mental instability. Valid
into subgroups such as male and female and younger and instruments are always reliable, too.
older. Then you estimate the proportion of people in each
subgroup (e.g., younger and older males and younger and
older females).
Reliability
Sample Size A reliable survey instrument is one that is relatively free
The size of the sample refers to the number of units that from measurement error. This error, caused individuals to
that will be surveyed to get precise and reliable findings. obtain scores that are different from their true scores,
The units can be people (e.g., boys and girls over and which can only be obtained from perfect measures.
under 16 years of age), places (e.g., counties, hospitals, What causes this error? In some cases, the error results
and schools), and things (e.g., medical or school records). from the measure itself – it may be difficult to understand
The number of needed units is influenced by a number of or poorly administered. For example, a self-administered
factors, including the purpose of the study, population questionnaire on the value of preventive healthcare might
size, the risk of selecting a bad sample, and the allowable produce unreliable results if its reading level is too high
sampling error (Kraemer and Thiemann, 1987; Cohen, for the teen mothers who are to use it. If the reading level
1988). When you increase the sample size, you increase is on target but the directions are unclear, the measure
the survey’s cost. Larger samples mean increased costs to will be unreliable. Of course, the survey researcher can
provide participants with financial or other incentives and simplify the language and clarify the directions and still
to follow up with nonresponders. find measurement error. This is because measurement
The most appropriate way to produce the right sample error can also come directly from the examinees. For
size is to use statistical calculations. These can be rela- example, if teen mothers are asked to complete a ques-
tively complex, depending on the needs of the survey. tionnaire and they are especially anxious or fatigued, their
Some surveys have just one sample, and others have obtained scores could differ from their true scores.
several. Formulas for calculating survey samples can be Four kinds of reliability are often discussed: stability,
found on the Internet. Type in the words, ‘‘sample size internal consistency, and inter- and intrarater reliability.
calculator.’’
Test–Retest Reliability or Stability
Response Rate A measure is stable if the correlation between scores from
one time to another is high. Suppose a survey of students’
All surveys hope for a high response rate. The response attitudes was administered to the same group of students
rate is the number that responds (numerator) divided by at School A in April and again in October. If the survey
158 Education Research Methodology: Quantitative Methods and Research

was reliable and no special program or intervention was sent to a clinic to observe waiting times, the appearance
introduced, then, on average, we would expect attitudes to of the waiting and examination rooms, and the general
remain the same. The major conceptual difficulty in estab- atmosphere. If the observers agreed perfectly on all items,
lishing test–retest reliability is in determining how much then interrater reliability would be perfect. Interrater reli-
time is permissible between the first and second adminis- ability is enhanced by training data collectors, providing
tration. If too much time elapses, external events might them with a guide for recording their observations, moni-
influence responses for the second administration; if too toring the quality of the data collection over time to see
little time passes, the respondents may remember and that people are not burning out, and offering a chance to
simply repeat their answers from the first administration. discuss difficult issues or problems. Intrarater reliability
When testing alternate-form reliability, the different refers to a single individual’s consistency of measurement,
forms may be administered at separate time points to and this, too, can be enhanced by training, monitoring, and
the same population. Alternatively, if the sample is large continuous education.
enough, it can be divided in half and each alternate form
administered to half the group. This technique, called the
split-halves method, is generally accepted as being as good
as administering the different forms to the same sample at Validity
different time points. When using the split-halves method,
you must make sure to select the half-samples randomly. Validity refers to the degree to which a survey instrument
assesses what it purports to measure. For example, a
survey of student attitude toward technological careers
Internal Consistency or Homogeneity would be an invalid measure if the survey only asked
about their knowledge of the newest advances in space
A survey’s internal consistency or homogeneity refers to technology. Similarly, an attitude survey will not be con-
the extent to which all the items or questions assess the sidered valid unless you can demonstrate that people who
same skill, characteristic, or quality. Cronbach’s coeffi- are identified as having a good attitude on the basis of
cient alpha, the average of all the correlations between their responses to the survey are different in some observ-
each item and the total score, is often calculated to deter- able way from people who are identified as dissatisfied.
mine the extent of homogeneity. For example, suppose a Four types of validity are often discussed: content, face,
survey researcher created a questionnaire to find out criterion, and construct.
about students’ satisfaction with Textbook A. An analysis
of homogeneity will tell the extent to which all items on
the questionnaire focus on satisfaction. Content Validity
Equivalence or alternate-form reliability is a type of
internal consistency. If two items measure the same con- Content validity refers to the extent to which a measure
cepts at the same level of difficulty, they are equivalent. thoroughly and appropriately assesses the skills or charac-
Suppose students were asked a question about their views teristics it is intended to measure. For example, a survey
toward technology before participating in a new computer researcher who is interested in developing a measure of
skills class and again 2 months after completing it. Unless mental health has to first define the concept (‘‘What is
the survey researcher was certain that the items on the mental health?’’ ‘‘How is health distinguished from disease?’’)
surveys were equal, more favorable views on technology and then write items that adequately contain all aspects of
after the second administration could reflect the survey’s the definition. The literature is often consulted either for
language level, for example, rather than improved views. a model or for a conceptual framework from which a
Some variables do not have a single dimension. Student definition can be derived because of the complexity of
satisfaction, for example, may consist of satisfaction with the task. It is not uncommon in establishing content valid-
school in general, their school in particular, teachers, clas- ity to see a statement like ‘‘We used XYZ cognitive theory
ses, extracurricular activities, and so on. If you are unsure to select items on mental health, and we adapted the ABC
of the number of dimensions expressed in an instrument, a role model paradigm for questions about social relations.’’
factor analysis can be performed. This statistical proce-
dure identifies factors or relationships among the items
or questions. Face Validity
Face validity refers to how a measure appears on the
surface: Does it seem to ask all the needed questions?
Inter- and Intrarater Reliability
Does it use the appropriate language and language level
Interrater reliability refers to the extent to which two to do so? Face validity, unlike content validity, does not
or more individuals agree. Suppose two individuals were rely on established theory for support.
 Survey Research Methods 159

Criterion Validity terms, identifying people who are compassionate and

those who are unfeeling (according to the theory), and
Criterion validity compares responses to future per-
proving that the measure consistently and correctly
formance or to those obtained from other, more well-
distinguishes between the two groups.
established surveys. Criterion validity is made up two
subcategories: predictive and concurrent. Predictive
validity refers to the extent to which a survey measure
forecasts future performance. A graduate school entry
examination that predicts who will do well in graduate Quality of Survey Data
school has predictive validity. Concurrent validity is
demonstrated when two assessments agree or a new mea- The quality of survey data depends on how well respon-
sure is compared favorably with one that is already con- dents understand the survey’s questions. Their compre-
sidered valid. For example, to establish the concurrent hension may be affected by language skills, education,
validity of a new survey, the survey researcher can either and culture. A pretest of the survey enables the survey
administer the new and validated measure to the same researcher to evaluate whether respondents reliably
group of respondents and compare the responses, or interpret questions and that they do so as they survey
administer the new instrument to the respondents and researcher intends. Usually small groups (five to ten people)
compare the responses to experts’ judgment. A high cor- who are similar in demographic and languages skills to
relation between the new survey and the criterion means prospective respondents evaluate each survey questions
concurrent validity. Establishing concurrent validity is individually or in a group. The idea is to decide whether
useful when a new measure is created that claims to be to accept the original question and meaning, to change the
better (shorter, cheaper, and fairer). question but keep the meaning, or to eliminate the ques-
tion or write a new one. Pretest participants are also asked
to examine the survey with regard to its flow and adminis-
trative ease, identifying unusual, redundant, irrelevant, or
Construct Validity poorly worded question stems and responses. They are
Construct validity is established experimentally to dem- also asked to record the time required to complete the
onstrate that a survey distinguishes between people who survey.
do and do not have certain characteristics. For example, a Pilot testing is a miniature version of an experiment in
survey researcher who claims constructive validity for which a sample of respondents is chosen to complete the
a measure of satisfaction will have to demonstrate in a survey for the purpose of evaluating its effectiveness in
scientific manner that satisfied respondents behave differ- practice. In theory all the technical problems that can be
ently from dissatisfied respondents. Construct validity is anticipated have already been resolved in the pretest.
commonly established in at least two ways: A pilot test is, however, the reality test. If the respondents
in the real setting are not as motivated as those in the
1. The survey researcher hypothesizes that the new mea- pretest environment, differ in their language or other
sure correlates with one or more measures of a similar skills, or are unfamiliar with surveys, the researcher may
characteristic (convergent validity) and does not corre- have to go back to the drawing board.
late with measures of dissimilar characteristics (discrim-
inant validity). For example, a survey researcher who
is validating a new quality-of-life survey might posit
that it is highly correlated with another quality-of-life Bibliography
instrument, a measure of functioning, and a measure of
health status. At the same time, the survey researcher Bourque, L. B. and Fielder, E. P. (2004). How to Conduct Telephone
Surveys. Thousand Oaks, CA: Sage.
would hypothesize that the new measure does not corre- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences,
late with selected measures of social desirability (the 2nd edn. New York: Academic Press.
tendency to answer questions so as to present yourself Fink, A. (ed.) (2004a). The Survey Kit. Thousand Oaks, CA: Sage.
Fink, A. (2004b). How to Ask Survey Questions. Thousand Oaks, CA:
in a more positive light) and of hostility. Sage.
2. The survey researcher hypothesizes that the measure Fink, A. (2008a). How to Conduct Surveys, 3rd edn. New Delhi: Sage.
can distinguish one group from the other on some Fink, A. (2008b). Practicing Research: Discovering Evidence that
Matters. New Delhi: Sage.
important variable. For example, a measure of compas- Jones, J. (1995). Qualitative research: Consensus methods for medical
sion should be able to demonstrate that people who are and health services research. British Medical Journal 311, 376–380.
high scorers are compassionate but that people who Kraemer, H. C. and Thiemann, S. (1987). How Many Subjects?
Statistical Power Analysis in Research. Thousand Oaks, CA: Sage.
are low scorers are unfeeling. This requires translating Krueger, R. A. and Casey, M. A. (2000). Focus Groups: A Practical
a theory of compassionate behavior into measurable Guide for Applied Research. Thousand Oaks, CA: Sage.
160 Education Research Methodology: Quantitative Methods and Research

Litwin, M. (2004). How to Assess and Interpret Survey Psychometrics. Relevant Websites
Thousand Oaks, CA: Sage.
Sudman, S. and Bradburn, N. M. (1982). Asking Questions. San
Francisco, CA: Jossey-Bass. http://www.bmj.com – BMJ Publishing Group Ltd.
http://www.eric.gov – Education Resources Information Center.
http://nces.ed.gov – The National Center for Education Statistics.
http://scientific.thomson.com – The Thomson Scientific Database.
Further Reading http://www.census.gov – US Census Bureau.

Campbell, D. T. and Stanley, J. C. (1963). Experimental and Quasi-

Experimental Designs for Research. Chicago, IL: Rand McNally.