Statistical Principles of
Experimental Design
Dov Stekel
Maximum information from
minimum effort
Overview
Blocking and randomization
Arrangement of samples and arrays
Class exercise
How many replicates?
Computer practical
Blocking, Randomization and
Blinding
Arrangement of experimental design that
minimises problems from extraneous
sources of variability
Use blocking to avoid confounding
Use randomization and blinding to avoid
bias
Toxicity Example
We are interested in characterising the
toxic effect of Benzo(a)pyrene (BP) on rats
8 Rats are to be treated with BP and 8 rats
with a control compound
Each array will be hybridized against a
reference sample
16 Arrays in the experiment
Experimental Design
There are two batches of 8 slides from two
different print runs (1 and 2)
Hybridisation will be done by two
researchers, Alison and Brian.
What is the best way to arrange the
experiment?
Design 1
Alison prepares all 8 BP samples and
hybridises them to the arrays of print run 1
Brian prepares all 8 control samples and
hybridises them to the arrays of print run 2
Design 2
Alison chooses 8 rats and treats 4 with BP and 4
with control substance.
She prepares and hybridises 2 BP samples to
arrays from print run 1 and 2 BP samples to
arrays from print run 2
She prepares and hybridises 2 control samples
to arrays from print run 1 and 2 control samples
to arrays from print run 2
Brian does the same with the other 8 rats
Design 2
Alison
Print Run 1
Print Run 2
Print Run 1
Print Run 2
Control
Treated
Brian
Control
Treated
Design 3
8 rats are randomly assigned to Alison, along
with 4 BP preparations and 4 control
preparations. She is not told which
preparations are which.
She prepares and hybridises samples to
randomly pre-arranged arrays so that 2 BP
samples and 2 control samples are hybridised
to 4 arrays from each of print runs 1 and 2.
Brian does the same with the other 8 rats
What is wrong with design 1?
Treatment, researcher and print run are
confounded variables
We cannot tell whether differences between the
two groups of rats result from treatment,
researcher or print run
Use blocking in designs 2 and 3 to deconfound
the variability of interest (treatment) from the
extraneous variabilities (researcher and print run)
Designs 2 and 3 are also balanced which
increases power of analyses
What is wrong with design 2?
Alison's choice of rats may be biased
For example, she may choose the
healthiest rats, so confounding potential
treatment effects with researcher variability
Use randomization and blinding in design
3 to avoid bias
Arrangement of Samples and Arrays
Is it better to use Affymetrix arrays or a
two-colour array system?
If using a two-colour array system, is it
better to use a reference sample?
If using a two-colour array system, what is
the best arrangement of samples on the
slides?
Several Factors
Available technology
Cost
Statistical considerations
We consider problem from perspective of
three different experiments
Example 1:
Hepatocellular Carcinomas
Samples are taken from disease and
healthy tissue from patients suffering from
hepatocellular carcinomas and hybridised
to microarrays. We would like to identify
genes that are up- or down- regulated in
hepatocellular carcinomas relative to
healthy tissue.
Design 1.1
Reference
Sample
Reference
Sample
Healthy 1
Disease 1
Array 1
Array 2
x 20
Design 1.2
Healthy 1
GeneChip 1
Disease 1
GeneChip 2
x 20
Design 1.3
Healthy 1
x 20
Disease 1
Array 1
Design 1.4
Healthy 1
Healthy 11
x 10
x 10
Disease 1
Disease 11
Array 1
Array 11
Design 1.5
Healthy 1
Healthy 1
x 20
Disease 1
Disease 1
Array 1
Array 2
Which is the best design?
Simple experiment - five different
designs!
Design 1.1 is bad because it increases
variability.
Design 1.3 is bad because it confounds
colour with disease state.
Designs 1.4 and 1.5 are best.
Design 1.1
Reference
Sample
Reference
Sample
Healthy
Disease
Array 1
Array 2
Coefficient of
Variability is 30%
Design increases
variability to 43%
Design 1.5
Healthy
Healthy
Disease
Disease
Array 1
Array 2
Coefficient of
Variability: 30%
Experimental
design reduces
variability to 21%
Example 2:
B-Cell Lymphomas
Samples are taken from 60 patients
suffering from B-cell lymphomas and
hybridised to microarrays. The aim of the
experiment is to identify clinically relevant
subgroups of patients using a cluster
analysis, and then to build a classification
model to differentiate between the
subgroups.
Design 2.1
Patient 1
x 30
Patient 2
Array 1
Design 2.2
Patient 1
x 60
Reference
Array 1
Design 2.3
Patient 1
GeneChip 1
x 60
Which design is best?
Design 2.1 is bad because it is difficult to
compare patients on equal footing.
Designs 2.2 and 2.3 are good.
Probably most appropriate use of
Affymetrix technology.
Example 3:
Yeast Time Series
Budding yeast can reproduce sexually by
producing haploid cells through a process
called sporulation. Yeast was placed in a
sporulating medium, samples taken at 7
timepoints from the start of sporulation.
We are interested in identifying genes that
show similar profiles in the timecourse.
Design 3.1
Time 0
Time 0
Time 0
Time 0
Time 0
Time 0
Time 1
Time 2
Time 3
Time 4
Time 5
Time 6
Array 1
Design 3.2
Time 0
Time 1
Time 2
Time 3
Time 4
Time 5
Time 6
Time 1
Time 2
Time 3
Time 4
Time 5
Time 6
Time 0
Array 1
Design 3.3
Time 0
Time 1
GeneChip 1 2
Time 2
Time 3
Time 4
Time 5
Time 6
Which is the best design?
Design 3.3 is bad because timepoint is
confounded with array.
Design 3.2 is a loop design. It is a good
design, but harder to analyse.
Design 3.1 is the best design.
Bright Timepoint Problem
Imagine we have a "bright" array. This
could be because of:
Higher gene expression
Experimental artifact
Normalising by array mean or median
cannot deconfound these factors
Time Series Example
Time Series Ratios
Raw Gene Expression for FYV1
FYV1 Normalised to Array
FYV1 Normalised to Reference
Class Exercise
Two strains of Staphylococcus aureus:
methicillin-sensitive and methicillinresistant
Each strain is cultured and then either
treated or untreated with methicillin
Samples are taken at several time points
(0h, 2h, 6h, 10h)
We want to identify genes involved in
methicillin-resistance
How Many Replicates?
Use Power Analysis which relates:
Difference in mean we are trying to detect
Population and experimental variability
Type of analysis
Chosen significance threshold
Number of replicates
Population Inferrence
Population
Sample
Inferrence
Confidence
The confidence is the probability of not getting
a false positive result.
It is the probability of accepting the null
hypothesis when the null hypothesis is true.
A false positive result is known as a Type I
Error.
We control for Type I errors explicitly by
selecting an appropriate confidence level
In microarray experiments, we must modify the
confidence level to account for multiplicity
Power
The power is the probability of not getting a false
negative result.
It is the probability of rejecting the null hypothesis
when the null hypothesis is false.
A false negative result is known as a Type II
Error.
We control the power implicitly via the confidence
level and the experimental design.
Type I and Type II Errors
TRUE SITUATION
OUR
DECISION
No effect
Effect
Not significant
Correct
Type II error
Significant
Type I error
Correct
Power Analysis Assumptions
We assume that the data is approximately log
normally distributed
This corresponds to standard deviation of the
errors of the raw data being proportional to the
signal intensity
This is equivalent to a constant standard
deviation in the logged data
The standard deviation divided by the mean is
called the coefficient of variation
Log Normally Distributed Data
Power Analysis
We will use the power.t.test() formula in
R to calculate the power of one and two
sample tests
power.t.test(n, delta, sd,
sig.level, power, type,
alternative)
Formula is used with one of the first five
variables omitted and will calculate the
unknown variable
Power Analysis Example:
Doxorubicin Chemotherapy
We are interested in the treatment of breast
cancer patients with doxorubicin chemotherapy
We want to perform a microarray experiment to
determine genes that are up- or down- regulated
as a result of the chemotherapy
We would like to know:
How to design the experiment?
How many patients we need?
Paired vs Unpaired Design
In a paired design, we take samples from each
patient before and after treatment, and for each
gene, look at the difference in expression before
and after treatment
In an unpaired design, we have two groups of
patients, one group treated, the other group
untreated. We look at the difference in gene
expression between the two groups
Which is a better experiment?
Paired and Unpaired Designs
Paired: test if
mean is different
from zero
Unpaired: test if
means of groups
are different
Power Analysis Assumptions
Suppose we know from a pilot study and
evaluation of our technology that the
coefficient of variation is 40%
Let's say that we want to detect genes that are
2-fold regulated
We are testing 10,000 genes so we will use a
signficance threshold of 0.001 to compensate
for multiplicity
How many patients do we need for a power of
80%, 90% and 99%?
Paired Experiment
The standard deviation of the underlying normal
distribution equivalent to 40% variability is 0.39
The difference in means is log2(2) = 1
The number of patients we need is:
Power
80%
90%
99%
Number
8
9
11
Unpaired Experiment
The standard deviation and difference in
means is the same.
The number of patients we need is:
Power
80%
90%
99%
Group Size
8
10
13
Number
16
20
26
1-Sample Number
8
9
11
Paired vs Unpaired
In this example, we need more than twice
the patients in the unpaired experiment to
obtain the same power as the paired
experiment
Paired experimental design is more
powerful than unpaired experimental
design because the differences between
individuals are factored out in the analysis
Conclusions
Extraneous variability:
Block to avoid confounding variables
Randomisation to avoid bias
Blocked experiments require ANOVA
analyses
Two sample experiments
Reference samples increase variability.
Hybridise both samples to same array.
Conclusions
Multiple patient comparisons
Reference samples or Affymetrix technology
enable comparisons.
Time series analysis
Reference samples are essential.
Number of replicates
Calculate using power analyses.
Computer Practical
Power analysis for population inference test
