Jack Bistritz

Psych 205: Research Methods

Book Notes

Chapter 1

A Brief History of Human Knowledge

 Barnum Descriptions: statements that are true of people in general, often perceived by

the individual to be true of them in particular

 Metaphysical Systems
o Metaphysical explanations are explanations that violate what scientists now

consider to be established physical laws, primarily by attributing behavior to

experiences to nonphysical forces, such as spirits or deities

o Animism—the belief that natural phenomena are alive and influence behavior
 Aristotle—belief that people possessing the physical attributes of certain

animals also possess the dispositions of those animals

o Mythology and religion—make the assumption that the deities play an important

role in human behavior

o Astrology—assumes that human behavior is determined by the activity of

celestial bodies
 Scientific properties—precision and accuracy in measurement
 Philosophy
o Philosophy is the study of knowledge, behavior, and the nature of reality by

making use of logic, intuition, and empirical observations

o Positivism—a principle based only on observations that can be made with

absolute certainty
o Empiricism—the idea that the best way to learn about the world is to make

observations
 Physiology and the Physical Sciences
o Physiology—the study of the functions of and interrelations between different

parts of the brain and body

  Introduction of the experimental method—powerful way to answer

research questions
o Experimental psychologists owe a great deal to the traditions and methods

developed and refined by physiologists

 Experimental Psychology
o Fechner, von Helmholtz, Weber, and Wundt

The Four Canons of Science

 Determinism
o The doctrine that the universe is orderly—that all events have meaningful,

systematic causes
o Base rate information is information about the proportion of things in a target

population—can be clouded by subjectively useful competing information

o Illusionary correlation—people falsely infer a connection or correlation between

group membership and the likelihood of engaging in nice vs. nasty behaviors
 Smaller groups tend to be less likeable
o Superstitious conditioning—the “false” conditioning that often occurs when an

organism is provided with reinforcements at random internals. It occurs because

the organism comes to associate an arbitrary response with the delivery of the

reinforcement and repeats the behavior until the next reinforcement

o A theory is a statement about the casual relation between two or more variables,

typically stated in abstract terms. Wouldn’t be useful in the absence of

determinism, because orderly, systematic causes wouldn’t exist.

 Empiricism
o The assumption that the universe obeys orderly principles, but that there are good

and bad ways of figuring out these orderly principles

o Best method is to make observations
 Parsimony
o If we are faced with two competing theories that do an equally good job of

handling a set of empirical observations, we should prefer the simpler of the two
 o Be extremely frugal in developing theories—steering away from unnecessary

concepts
 Testability
o Assumption that scientific theories should be testable using research techniques
o Falsifiability—scientists should go a step beyond by actively seeking out tests

that could prove their theories wrong

o Popper and logical positivism
 Belief that science and philosophy should be based solely on things that

can be observed with absolute certainty. Actively try to disprove them

o Operational definitions—definitions of theoretical constructs that are stated in

terms of concrete, observable procedures

 Connecting unobservable traits/experiences to things that can be observed
 “Hunger” in terms of hours of food deprivation
 Used in scoring for all sports—would never be any winners or losers

without operational definitions

Four Ways of Knowing About the World

 Intuition
 Logic
 Authority
 Observation
 There is no guarantee that one way of knowing will be superior to others across all

possible situations

Chapter 2: How Do We Find Out?

Laws, Theories, and Hypotheses

 Psychologists are in the business of seeking out laws that allow them to make precise

predictions of human behavior

 A law is a universal statement of the nature of things that allows reliable predictions of

future events
 A theory is a general statement about the relation between two or more variables
o Good theories share all the features of good science—should be logical and

orderly, emphasizing the systematic causal relations between variables

  Generate predictions about readily observable events
 Parsimonious—simple and concise
 Testable
 Difference between laws and theories
o Laws are comprehensive, fundamental statements about reality
o Theories have boundary conditions; there are plenty of times they would not

apply—only offer accurate prediction in certain circumstances

o Two or more correct theories will sometimes cancel one another out
 The principle of equifinality—the notion that the same behavior is often produced by

many different causes

o Can cause psychologists to disagree about the conditions favoring one theory over

another, so good theory testing will often take the form of determining each

theory’s proper domain of application

 Hypotheses are predictions about specific events that are derived from one of more

theories

The Science of Observation

 Method of induction—making many observations under controlled conditions and

arriving at a general statement about how things are

o Induction—reasoning from specific instances to general principles
o If new observations are consistent with the statement, the statement survives. If

not, the statement is either discarded or revised

 If the statement is revised it is tested against new observations and the

process starts over again—eventually it becomes so precise it becomes a

law
 Hume’s problem of induction
o How do you know when you’ve made enough observations to be sure that your

law is true? According to Hume you never do—there is always possible that the

very next observation will prove you wrong

o Fundamental attribution error—people’s tendency to favor dispositional over

situational explanations
 Miller’s findings in India disprove this theory
  Method of deduction
o Deduction refers to reasoning from the general to the specific
o Occurs when a general statement (theory) is used to develop predictions

(hypotheses) that are then tested against observations

 Why have theories/laws if they can never be proved true?
o If we are willingly to live with a little bit of uncertainty and trust science to

correct itself, we don’t need to worry about the fact that very few statements are

ever completely true

o A great deal may be learned from failure—can be more useful to learn what is not

true, than what is true

Three Approaches to Hypothesis Testing

 Positive test bias—tendency for people who are evaluating hypotheses to attempt to

confirm rather than disconfirm these hypotheses

o Behavioral confirmation—the tendency for social perceivers to elicit behaviors

from a person that are consistent with their initial expectancies of the person
o Once we get an idea in our heads, most of us tend to engage in hypothesis-

confirming behaviors that may falsely convince us the ideas are correct
 Once we have been exposed to some tentative evidence in support of our

theories or ideas, we become very reluctant to give them up

 The three general approaches when testing hypothesis include validation, falsification,

and qualification

Validation

 Validation is an approach to hypothesis testing in which researchers attempt to

gather evidence that supports or confirms a theory or hypothesis

 Problems:
o Researchers make implicit choices about what kind of data to examine, and

may even engineer laboratory situations that are highly conducive to

supporting that theory

  Cognitive dissonance test with the $1 vs. $20 reward—practicing a validation

approach
 Some researchers have argued that psychology has been plagued by the worst form

of validation—referred to as the replication crisis, or the publication of results that

cannot be replicated

Falsification

 Falsification is an approach to hypothesis testing in which researchers attempt to

gather evidence that invalidates or disconfirms a theory or hypothesis

o Psychologists want to believe that their theories are truthful, so not

practiced as much as it should be

o However, any study that involves careful, objective data collection can

yield results that disprove a hypothesis

o Theories are also open to scrutiny from a world of other researchers
o Daryl Bem and the falsification of the cognitive dissonance test
 Self-perception theory more parsimonious—does not require

elaborate assumptions about aversive states of arousal

Qualification

 Qualification is an approach to hypothesis testing in which researchers try to

identify the boundary conditions under which a theory or hypothesis is and isn’t

true
o Can lead to the integration of two contradictory theories by specifying the

conditions under which each of the theories is correct

o Confirmation bias can be present in developing a specific pattern of

results in qualification
o Researchers who appreciate the merits of more than one theory

sometimes to bridge the gap between validation and falsification by trying

to figure out exactly when each of a given set of competing theories is

correct
  Experimental paradigm: the approach to research in which the researcher

randomly assigns participants to different treatment conditions, measures some

outcome of interest, and makes use of inferential statistical tests to draw

conclusions about the effects of the manipulation

The Art of Scientific Discovery

 It is important to have an interesting topic to study

 Inductive Techniques for Developing Ideas
o The inductive techniques McGuire identified are all based on some kind of

specific observation
o Case studies—carefully documented observations of a specific group or person
o Trying to account for paradoxical incidents
 Puzzling or nonsensical observations
 Ex. Why people continue to gamble when the long-term effect is

loss, why people stay in an abusive relationship

o Analyzing the practitioner’s rule of thumb
 Analyzing things that experts in a particular area do to achieve certain

outcomes
o B.F Skinner notes that serendipity (luck or good fortune) plays an important role

in most big discoveries

 Partial reinforcement effect—responses more resistant to extinction when

they are reinforced inconsistently rather than uniformly

 Deductive Techniques for Developing Ideas
o Reasoning by analogy: analyzing similarities or concordances between different

phenomena to shed light on the less well understood of the two phenomena
o Applying a functional or adaptive analysis to a particular research question
 Researchers who adopt this strategy ask themselves basic questions about

what organisms have to do to master their environments

o Hypothetico-deductive model
 Hull (1943) believed a good way to go about research is to begin with a

set of basic assumptions and to derive one or more logical consequences

from these basic principles

 o Accounting for conflicting results
 Attempting to come up with theoretical reasons why different studies on

the same topic have yielded different findings

o Accounting for exceptions
 Attempting to generate exceptions or limiting conditions to well-

established psychological principles or empirical findings

The Ethics of Scientific Discovery

• The Evolution of Ethical Guidelines

o 1932 syphilis study—research team ordered doctors not to treat patients with

syphilis, even after known cure was developed

 The start of APA ethical standards and guidelines
 First set of standards published in 1958
o Freedom from coercion is a fundamental ethical protection
 Should never have to continue to participate in a study that makes you

uncomfortable
o Milgram’s controversial study—conducted extensive debriefing sessions
 Setting the record straight in a deception study after the study is done
o Now there are committees whose job it is to review research methods
• Modern Internal Review Boards and Risk-Benefit Analyses
o The ethics of all psychological studies are examined under an umbrella principle

known as a risk-benefit analysis

 Some greater good must come from psychologists’ use of human subjects

in their studies
 At the least, subjects should get something back for participating
o All universities and other research institutes must maintain an internal review

board
 They perform risk-benefit analyses to ensure all studies meet consensual

community standards for ethical behavior

 Some studies may be exempted from a full review if they do not make use

of deception or pose meaningful risks to participants

o Informed Consent
 Participants must be told about any potential risks—however slight
o Protection from physical and psychological harm
o Confidentiality
o Debriefing
Chapter 3: An Overview of Psychological Research Methods

• Risk-benefit rule—must be some sort of benefits to society to offset any risks to

participants. Two ways to ensure these benefits:

o Choose a research problem that matters
o Research must tell us something—must be methodologically rigorous
• Research is almost always informative if it maximizes
o Internal validity
 Information about what causes what
o External validity
 Information about how well a research finding holds up in the real world

Three Requirements for Establishing Causality

• Covariation
o For one variable to cause another, changes in one variable must correspond with

another
 If an increase in testosterone does not correlate with an increase in

aggression it’s pretty hard to support that theory

o Cannot easily make a claim about causation in the absence of covariation
o Covariation in itself is not enough to establish causality
• Temporal Sequence
o To argue that changes in one variable cause changes in a second—one must be

able to show that the changes in the first variable preceded the changes in the

second
o In a passive observational research design (cross-sectional, measuring a wide

range of variables at the same time, no experimental manipulation) it’s often

impossible to establish temporal sequence

o In correlation studies, it’s hard to tell what caused what
o In prospective designs researchers measure all of the variables of interest on at

least two different occasions

 Thus, can see if changes in one variable do proceed changes in a second
• Eliminating Confounds
o Must rule out all of the competing causes of an outcome that happen to be

correlated with the cause we think we’ve identified

 o Third-variable problem: two variables covary with one another and give the

false appearance of a causal relation

• The Magic of Random Assignment
o Create two identical groups of research participants and study them in a true

experiment
 Variable that is manipulated= IV, Variable that is measured=DV
o Key to eliminating confounds is random assignment
 Works best when you have a large sample
 Placing people in different conditions in an experiment on a totally

arbitrary basis

Passive Observational Research Methods

• Measuring naturally existing variation in the variables they are interested in

• Surveys and interviews
o Focus on real world behavior
o Convenience sample—researchers must be content with the small group of

people that is readily available for the survey

o Random sampling is the ideal way to sample people in
 Picking people at random from a target population
o Selection bias—Sampling people in a way that your participants do not represent

the population you were hoping to measure well

o 2 problems with surveying
 People are not always about to report their experiences honestly
 Fallible memories or language barriers
 People are not always willing to report their experiences honestly
 Ex. Social desirability/legal worries
o Ways to eliminate these problems
 Be sensitive to the issue of time
 Confidentially—ensuring participants know their answers are private
 Bogus pipeline—fake lie detector test created to give participants the

impression that the experimenter can detect subtle movements in their

hands that will reveal their true attitudes

 Tricks participants into being more honest, especially on tougher

judgment questions
 Fake lie detectors work best when you ask people what the

machine will say about their attitudes

 o People are more honest when they are filling out surveys instead of answering

questions face to face

 Putting a mirror in the room can increase honesty
• Unobtrusive Observation
o Recording people’s behavior in response to sensitive subject matter when they

don’t know you are doing so

o Observations are only unobtrusive if
 Researchers don’t interfere in any way with people’s natural behavior
 Research participants don’t have any idea they are being observed
o Examples:
 Garbology—Searching through people’s trash before asking what they

buy
 Google Correlate—looking through people’s search history
 A tool that can help in this method
• Archival Research
o Uses existing public records to test research hypotheses
• Ethnographies
o A narrative that describes a culture or a part of a culture
o Require a lot of behavioral coding
o Observes one small group

Trade-Offs Between Internal and External Validity

GAGES: The “Big Give” Of Worrisome Confounds

• Geography, Age, Gender, Ethnicity, SES (socioeconomic status)

• Geography
o Location matters—the environment that someone grows up in provides a lot of

detail about how they lived their life.

• Age
o Many differences in older and younger Americans (older eat healthier, worry less,

exercise less, etc.)

• Gender
o Influences not only how we behave, but how we are treated by others
• Ethnicity
o Influences our trust in police, depression level, likelihood of being bilingual or an

immigrant, etc.
• Socioeconomic Standing
o Predicts important attitudes and values
 • Reverse confound—a confound that makes it harder than it would be otherwise to

observe an effect

External Validity and the Oops! Heuristic

• Useful to have rules of thumb to analyze external validity:

• Operationalizations
o Questions about operational definitions—defining clear and precise operational

definitions of those things

o Converting your research idea into measurable actions and procedures
• Occasions
o Questions on generalization based on time—time matters! Have to ask ourselves

if that finding would hold true at other times

• Populations
o Research findings become more impressive if they hold up against multiple

different populations
• Situations
o The situation in which experiments are presented in can vary results (ex. casual

feeling vs. formal feeling effect our thought process)

o Need to know how well it holds up across different situations
• Are GAGES and OOPS! WEIRD?
o Most psychology focuses on “Western, educated, industrialized, rich, and

democratic” societies
 WEIRD focuses on cultures, whereas OOPS focuses on individuals
o Is there a good way to maximize external validity while also minimizing threats to

internal validity?
 Conduct true experiments that work hard to consider to OOPS heuristic
 Studying two diverse populations
 Capitalizing on a single, very diverse population

Maximizing External Validity in the Lab

• Zajonc’s mere exposure effect—the finding that the more often people are exposed to

something, the more they usually like it

o Language experiment with made up Turkish words confirmed his theory (people

liked the words they were more often exposed to)

o Held true in all the OOPS confounds
Gauging Gages in Archival Studies of Social Cognition

• False Consensus
o Brian Mullen—false consensus effect
 Tendency for people to overestimate the percentage of others who share

their beliefs or behaviors

 Larger if in the statistical minority for that belief
• Ethnic Stereotyping and Discrimination
o Study showing how more stereotypical looking black men were more likely to

receive the death penalty in response to murdering a white man

 When the murder was another black man the stereotypically black features

did not play a part

• Counterfactual Thinking and Emotion
o Counterfactual thinking shows that when something good or bad happens

people often consider alternative realities

 Bronze medal usually more pleased with their performance than silver

medalist

Chapter 4: Making it Happen

Step 1: Hypothesis Generation

• IDEA heuristic: integration, dissection, extension, and application

o Integration
 A hypothesis that is designed to pit two opposing theories, ideas, or

findings against each other

o Dissecting
 Pulling a single theory apart, to see exactly what it means
 Dig a little deeper to figure out more precisely why something is

true (usually a more generalized theory)

 Can be a solution to problems of scenario studies
 Studies that manipulate a variable by asking some people what

they’d do in one particular situation and asking another group what

they’d do in a variation on that situation

o Extension
  Test the limits of an established theory by considering the four aspects of

the OOPS! Heuristic

o Application
 Researchers who conduct applied research begin with a basic research

finding
 Then ask themselves if this is a real-world problem that can be

better understood through the lens of the basic research finding in

question
 Focus is on important social problems

Step 2: Operationalization (Design)

• Transforming the research idea into a testable hypothesis

• Read the Literature
o Don’t try to reinvent operational definitions for psychological constructs—do

your research on what has worked best/is most established in history

 Careful work has already been done on it
• Consult an Expert
• Maximize Statistical Power
o Power is a study’s ability to detect a true effect
 Bigger sample sizes typically yield more power
 Create dependent variables that are continuous rather than categorical
 Continuous variables can take on many values from very low to

very high
• Keep it Simple
o Keeping your research design simple usually means any necessary statistical

analyses of your findings will also be much simpler

• Make Sure You Have a Design
o A framework that lets you make a clear empirical statement
o Two-groups design
 Two or more groups receive different levels of your IV yielding

continuous scores on one or more dependent variables

o A study in which neither the participants nor the experimenters know which

treatment people are receiving is known as a double-blind procedure

• Consider Attention Checks
 o Include a couple of attention checks to make sure your participants are really

reading or listening to your instructions

• Collect Some Pilot Data
o Pilot study = a practice study that is often a simplified version of the real thing
 Can yield preliminary information about important issues
 Are instructions clear? Is there variation on your DV?

Step 3: Permission

• Exemptions and Expedited Reviews

o Some categories of research qualify for exemption of approval or expedition
 Low risks to participants

Step 4: Execution/Data Collection

Step 5: Calculation/Data Analysis

Step 6: Communication

Chapter 5: Validity, Reliability, and Measurement

Validity

• Validity refers to the relative accuracy of the statement

o Internal Validity
 The extent to which a set of research findings provides compelling

information about causality

 High in internal validity= confidence that variations in the IV caused any

observed changes in the DV

 Great for controlled lab experiences because of random assignment

and isolated manipulation

o External Validity
 The extent to which a set of research findings provides an accurate

description of what happens in the real world

 Can never be perfect—life is complex enough that enough the hardiest

research fails to generalize to every imaginable sample or situation

  Closely related to boundary conditions—when the boundary conditions

of a specific research finding are very narrow, this finding is low in

external validity
o Construct Validity
 The extent to which the IVs and DV truly represent the hypothetical

variables of interest to the research team

 AKA a direct reflection of the quality of a researcher’s operational

definitions
o Conceptual Validity
 How well a specific research hypothesis maps onto the broader theory that

it was designed to test

Reliability

• Reliability is the consistency or repeatability of a measure or observation

• How do we test reliability?
o Test-retest reliability
 Testing a group of participants at one time and having them come back a

second time to take the test again

 More impressive after a long period of time (temporal consistency)
 Two-four week gap is generally the standard (cannot wait too long

or people will change their opinions as they change as people)

 Internal consistency
 The degree to which the total set of items or observations in a

multiple-item behave in the same way

o Ex. The degree to which each item in a ten-item self-esteem

inventory rank orders participants in the same manner

 Interobserver agreement/interrater reliability
 Refers to the degree to which different judges independently agree

upon an observation
 The ratings of multiple judges are useful only if they are made by

trained and independent judges

 Most helpful when measuring a behavior that cannot easily be

assessed
Reliability, Validity, and the “More is Better” Rule

• The logic of the “more is better” rule applies very widely in psychological science
• Test-retest correlation: assessing temporal stability by taking a measurement on two

different occasions and correlating people’s scores at time 1 with their scores at time 2
• Most forms of reliability can be assessed statistically, most forms of validity require

logical analyses

Measurement Scales

• Nominal Scales
o Nominal (or categorical) scales involve meaningful, but non-numerical/categories
 Ex. Gender
• Ordinal Scales
o Involve order or ranking
 Ex. Birth order/ranking in a contest
o Not great at giving us absolute differences between subjects
• Interval Scales
o Measurement scales that make use of real numbers designating amounts to reflect

relative differences in magnitude

o Can take on negative values
o Corresponds to a specific amount of the construct being measured
• Ratio Scales
o Much like interval scales except that they always have a true zero point
 A point at which none of the quantity under consideration is present
• The Validity of Measurement Assumptions
o It’s not possible to create a ratio scale merely by starting a scale’s lower endpoint

at 0 and moving upward in small increments that feel equal to you

Chapter 6: Moving from Notions to Numbers

Converting Notions to Numbers: The Two Major Challenges

• 1. Ensuring that research participants are thinking about the same question that the

researcher was thinking about

o Judgement phase
• 2. Ensuring that participants are able to translate their internal psychological state into

some kind of value on a response scale

o Response translation page
• The Judgement Phase
 o During the judgement phase participants determine what question is being asked,

and they form an initial response to that question

o Need to use language that participants will most easily understand
o Pilot testing—use of practice studies that are designed to help researchers refine

the measures or manipulations they wish to use in the real study

o Focus groups—a small but representative sample of participants a researcher

wishes to understand meet together to discuss their experience

 Answer some open-ended questions
• Word Questions Well for Everyone: Being Clear and Simple
o Keep it Simple
o Use Informal Language
o Avoid Negations
o Avoid Double-Barreled Questions: a question that asks you to evaluate two

different things using a single response.

o Avoid Forced-choice questions: questions that ask participants to select one of

two or more options

o Avoid Questions that do not yield any variance
 Avoid floor effects and ceiling effects
 Floor effects occur when almost everyone in a sample responds to

the same low level on a question

 Ceiling effects= when almost everyone responds at the high level
o Avoid loaded questions: write questions in ways that do not indicate which

response the researcher considers most desirable

o Make sure your questions are relevant to everyone in your study
o Write multiple questions to assess the same construct
o Mix it up
 Make use of both positively worded and negatively worded questions
o Establish a judgmental context
 Researchers should establish an appropriate context
o Ease into socially sensitive questions
o Ask sensitive questions sensitively
o Guarantee Participants’ Anonymity

The Response Translation Phase

• The number of scale points

o Can’t be too large or too small
 Intermediate numbers important
  3-10
• The Importance of Anchors
o Adjectives that lend meaning to the numbers on a scale
o Choosing anchors to make equal-appearing intervals
• Unipolar vs. Bipolar scales
o Bipolar scales ask participants to rate a quantity that deviates in both directions

from a zero point

• Semantic Scales, Guttman Scales, and Thurston Scales