Research Methods

Lecture 10

Words of Wisdom
•There are no permanent failures, only
temporary setbacks.
•All the problems and setbacks com to
teach us something. They do have
answers. Our job is to find those
answers.
• So verily with every difficulty there is relief. Verily with every
hardship there is relief. Al-Qur’an (94:5-6).

1
 What Is a Case Study?
• A case study is an empirical method
– A defined, scientific, method for posing research
questions, collecting data, analyzing the data, and
presenting the results.
• A case study is an empirical inquiry that
– Investigates a contemporary phenomenon within its real-
life context, especially when
– the boundaries between phenomenon and context are
not cleraly evident
• Case studies can be used for validating research
results

When should you use a case

study?
• When you can’t control the variables
• When there are many more variables than data points
• When you cannot separate phenomena from context
– Phenomena that don’t occur in a lab setting
– E.g. large scale, complex software projects
– Effects can be wide-ranging
– Effects can take a long time to appear (weeks, months, years!)

• When the context is important

– E.g. When you need to know how context affects the phenomena

• When you need to know whether your theory applies to a

specific real world setting

2
 Why conduct a case study?
• To gain a deep understanding of a phenomenon
– Example: To understand the capability of a new tool
– Example: To identify factors affecting communication in code
inspection
– Example: To characterize the process of coming up to speed on a
project
• Objective of Investigation
– Exploration- To find what’s out there
– Characterization- To more fully describe
– Validation- To find out whether a theory/hypothesis is true
• Subject of Investigation
– An intervention, e.g. tool, technique, method, approach to design,
implementation, or organizational structure
– An existing thing or process, e.g. a team, releases, defects

Misuses of the term “Case

Study”
• Not a case history
– In medicine and law, patients or clients are “cases.” Hence
sometimes they refer to a review of interesting instance(s) as a
“case study”.
• Not an exemplar
– Not a report of something interesting that was tried on a toy
problem
• Not an experience report
– Retrospective report on an experience (typically, industrial) with
lessons learned
• Not a quasi-experiment with small n
– Weaker form of experiment with a small sample size
– Uses a different logic for designing the study and for generalizing
from results

3
 How can I tell it’s a case study?
• Has research questions set out from the beginning
of the study
• Data is collected in a planned and consistent
manner
• Inferences are made from the data to answer the
research questions
• Produces an explanation, description, or causal
analysis of a phenomenon
– Can also be exploratory
• Threats to validity are addressed in a systematic
way

Parts of a Case Study Research

Design
• A research design is a “blueprint” for a study
• Deals more with the logic of the study than the logistics
• Plan for moving from questions to answers
• Ensures data is collected and analyzed to produce an
answer to the initial research question
• (Analogy: research design is like a system design)
• Five parts of a case study research design
– 1. Research questions
– 2. Propositions (if any)
– 3. Unit(s) of analysis
– 4. Logic linking the data to the propositions
– 5. Criteria for interpreting the findings

4
 Study Question
• Study design always starts with research questions
• Clarify precisely the nature of the research question
• Ensure the questions can be answered with a case study

• Generally, should be “how” and “why” questions.

• Identify and interpret the relevant theoretical constructs

• Examples:
– “Why do two organizations have a collaborative relationship?”
– "Why do developers prefer this tool/model/notation?"
– "How are inspections carried out in practice?“
– "How does agile development work in practice?"
– "Why do programmers fail to document their code?“
– "How does software evolve over time?“
– "Why have formal methods not been adopted widely for safety-
critical software?“

Types of Case Study

• Explanatory
– Adjudicates between competing explanations (theories)
– E.g. How important is implementation bias in requirements
engineering?
• Rival theories: existing architectures are useful for anchoring, vs.
existing architectures are over-constraining during RE
• Descriptive
– Describes sequence of events and underlying mechanisms
– E.g. How does pair programming actually work?
– E.g. How do software immigrants naturalize?
• Causal
– Looks for causal relationship between concepts
– E.g. How do requirements errors and programming errors affect
safety in real time control systems?
• Exploratory
– Used to build new theories where we don’t have any yet
– Choose cases that meet particular criteria or parameters
– E.g. What do CMM level 3 organizations have in common?

5
 Study Propositions
• Propositions are claims about the research question
– State what you expect to show in the study
– Direct attention to things that should be examined in the case study
– E.g. “Organizations collaborate because they derive mutual
benefits”
• Propositions will tell you where to look for relevant evidence
– Example: Define and ascertain the specific benefits to each
organization
• Note: exploratory studies might not have propositions
– …but should lead to propositions for further study
– …and should still have a clearly-stated purpose and clearly-stated
criteria for success

• Analogy: hypotheses in controlled experiments

Unit of Analysis
• Defines what a “case” is in the case study
– Choice depends on the primary research questions
– Choice affects decisions on data collection and analysis
– Hard to change the unit of analysis once the study has started (but
can be done if there are compelling reasons)
– Note: good idea to use same unit of analysis as previous studies
(why?)
• Often many choices - E.g. for an exploratory study of
extreme programming:
– Unit of analysis = individual developer (case study focuses on a
person’s participation in the project)
– Unit of analysis = a team (case study focuses on team activities)
– Unit of analysis = a decision (case study focuses on activities
around that decision)
– Unit of analysis = a process (e.g. case study examines how user
stories are collected and prioritized)

6
 Examples of Unit of Analysis
• For a study of how software immigrants naturalize
– Individuals?
– … or the Development team?
– … or the Organization?
• For a study of pair programming
– Programming episodes?
– … or Pairs of programmers?
– … or the Development team?
– … or the Organization?
• For a study of software evolution
– A Modification report?
– … or a File?
– … or a System?
– … or a Release?
– … or a Stable release?

Why Defining your Unit of

Analysis matters
• Clearly bounds the case study
– …and tells you which data to collect
• Makes it easier to compare case studies
– …incomparable unless you know the units of analysis
are the same
• Avoid subjective judgment of scope:
– E.g. disagreement about the beginning and end points of
a process
• Avoids mistakes in inferences from the data
– E.g. If your study proposition talks about team
homogeneity…
– …Won’t be able to say much if your units of analysis are
individuals

7
 Linking data to propositions
• One of the least well developed components in
case studies
• Pattern Matching
– Describe several potential patterns, then compare the
case study data to the patterns and see which one is
closer
• Which pattern matches best?

”effects” pattern ”no effects” pattern Observation

Interpretation Criteria
• Criteria for interpreting a study’s findings
– i.e. before you start, know how you will interpret your findings
• Also a relatively undeveloped component in case studies
– Currently there is no general consensus on criteria for interpreting
case study findings
– [Compare with standard statistical tests for controlled experiments]
• How close does a match have do be in order to be
considered a match?
• No general solution…
• Hope that patterns of rival propositions are sufficiently
constrasting

8
 Generalization
• Statistical vs. Analytical

• Quantitative methods tend to sample over a

population
• Statistical tests then used to generalize to the
whole population

• Qualitative methods cannot use statistical

generalization
• Hence use analytical generalization

Generalization
• Statistical • Analytical
• First level generalization: • Second level generalization:
– From sample to population – From findings to theory
– Well understood and widely • Compares qualitative findings
used in empirical studies with the theory:
• Can only be used for – Does the data support or refute
quantifiable variables the theory?
• Based on random sampling: – Or: do they support this theory
– Standard statistical tests tell better than rival theories?
you results on a sample apply to • Supports empirical induction:
the whole population if – Evidence builds if subsequent
• Not useful for case studies case studies also support the
– No random sampling theory (& fail to support rival
theories)
– Rarely enough data points
• More powerful than statistical
techniques
– Doesn’t rely on correlations
– Examines underlying
mechanisms

9
Q&A

10