C LUSTERING IN

B IOINFORMATICS

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 O VERVIEW

Define the clustering problem

Motivation: gene expression and microarrays

Types of clustering

Clustering algorithms

Other applications of clustering

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 T HE CLUSTERING PROBLEM
Motivation: Find patterns in a sea of data

Input:

A (large) number of datapoints: N

A measure of distance between any two data points dij

Output:

Groupings (clustering) of the elements into K (the number can be user-

specified or automatically determined) ‘similarity’ classes

Sometimes there is also an objective measure that the obtained clustering

seeks to minimize.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 A MARQUEE APPLICATION :
M ICROARRAY ANALYSIS
What do newly sequenced genes do?

Simply comparing the new gene sequences to known DNA sequences often does not necessarily reveal
the function of a gene: for 40% of sequenced genes, functionality cannot be ascertained by only
comparing to sequences of other known genes

Genes that perform similar or complementary function to known genes (reference) will be expressed
(transcribed) at high levels together with known genes

Genes that perform antagonistic functions (e.g. down-regulation) may be expressed at high levels at an
earlier or later time point when compared to known genes

E.g. what happens to gene expression in cancer cells?

Expression level is estimated by measuring the amount of mRNA for that particular gene

A gene is active if it is being transcribed

More mRNA usually indicates more gene activity

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 A MICROARRAY EXPERIMENT
Produce cDNA from mRNA (cDNA is more stable)

Label cDNA with a fluorescent dye or biotin for detection

Different color labels are available to compare many samples at once

Wash cDNA over the microarray containing thousands of high density probes
that hybridize to complementary strands in the sample and immobilize them on
the surface.

For biotin-labeled samples, stain with the biotin-specific fluorescently labeled

antibody

Read the microarray, using a laser or a high-resolution CCD

Illumination reveals transcribed/co-expressed genes

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 HTTP://UPLOAD.WIKIMEDIA.ORG/WIKIPEDIA/COMMONS/0/0E/MICROARRAY2.GIF

Green: expressed only in control

Red: expressed only in an experimental cell

Yellow: equally expressed in both samples

Black: NOT expressed in either control or sample

HTTP://UPLOAD.WIKIMEDIA.ORG/WIKIPEDIA/EN/C/C8/MICROARRAY-SCHEMA.JPG

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Track the sample over a period
of time to observe changes in
gene expression over time

Track two samples under the

same conditions to look for
differential expression

Each box represents one gene’s

expression over time

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 M ICROARRAY D ATA

Microarray data are usually transformed into a (relative, normalized)

intensity matrix

Can also be represented as a bit matrix (log2 of relative intensity)

The intensity matrix allows biologists to infer correlations between

different genes (even if they are dissimilar) and to understand how
genes functions might be related

Care must be taken to normalize the data appropriately, e.g.

different time points can come from different arrays.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Gene Time 1 Time 2 Time 3
1 10 8 10
2 10 0 9 Which genes are similar?
3 4 8.5 3
4 9.5 0.5 8.5
5 4.5 8.5 3
What defines co-expression?
6 10.5 9 12
7 5 8.5 11 How to measure the distance/
8 2.7 8.7 2 similarity?
9 9.7 2 9
10 10.2 1 9.2

INTENSITY TABLE

EUCLIDEAN DISTANCE IN D-DIMENSIONS

�
� d
��
D(x, y) = � (x − y )2 i i
i=1

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 F INDING SIMILAR GENES
1 2 3 4 5 6 7 8 9 10

1 8.1 9.2 7.7 8.9 2.3 5.1 10.9 6.1 7.0

2 8.1 12.0 0.9 11.8 9.5 10.1 13.3 2.0 1.0

2,4,9,10 10 3 9.2 12.0 11.2 0.5 11.1 8.1 1.7 10.5 11.5

1,6,7 4 7.7 0.9 11.2 10.9 9.2 9.5 12.5 1.6 1.1

5 8.9 11.8 0.5 10.9 10.8 8.0 2.1 10.3 11.3

6 2.3 9.5 11.1 9.2 10.8 5.6 12.7 7.7 8.5

7 5.1 10.1 8.1 9.5 8.0 5.6 9.3 8.3 9.3

10
5 8 10.9 13.3 1.7 12.5 2.1 12.7 9.3 12.0 12.9

9 6.1 2.0 10.5 1.6 10.3 7.7 8.3 12.0 1.1

10 7.0 1.0 11.5 1.1 11.3 8.5 9.3 12.9 1.1

5 0 PAIRWISE DISTANCES
10 1 6 7 2 4 9 10 3 5 8

3,5,8 1 0.0 2.3 5.1 8.1 7.7 6.1 7.0 9.2 8.9 10.9

6 2.3 0.0 5.6 9.5 9.2 7.7 8.5 11.1 10.8 12.7
5
0 7 5.1 5.6 0.0 10.1 9.5 8.3 9.3 8.1 8.0 9.3
0 2 8.1 9.5 10.1 0.0 0.9 2.0 1.0 12.0 11.8 13.3
5
10 0 4 7.7 9.2 9.5 0.9 0.0 1.6 1.1 11.2 10.9 12.5

9 6.1 7.7 8.3 2.0 1.6 0.0 1.1 10.5 10.3 12.0

10 7.0 8.5 9.3 1.0 1.1 1.1 0.0 11.5 11.3 12.9

3 9.2 11.1 8.1 12.0 11.2 10.5 11.5 0.0 0.5 1.7

6 8.9 10.8 8.0 11.8 10.9 10.3 11.3 0.5 0.0 2.1

8 10.9 12.7 9.3 13.3 12.5 12.0 12.9 1.7 2.1 0.0

REARRANGED DISTANCES
CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]
 C LUSTERING P RINCIPLES
Homogeneity: elements of the same cluster are maximally close to
each other

Separation: elements in separate clusters are maximally far apart

from each other

One is actually implied by the other (in many cases)

Generally speaking, this is a hard problem.

 
� �
min α d(x, y) − β d(x, y)
clustering
x,y∈the same cluster x,y∈different clusters

RELATIVE IMPORTANCE
CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]
 BECAUSE
� �
d(x, y) + d(x, y)
x,y∈the same cluster x,y∈different clusters
�
= d(x, y) = D = const
x,y
WE CAN
 SIMPLIFY 
� �
min α d(x, y) − β d(x, y)
clustering
x,y∈the same cluster x,y∈different clusters

TO AN EQUIVALENT EXPRESSION THAT ONLY DEPENDS ON

INTRA-CLUSTER DISTANCES

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 P OOR CLUSTERING EXAMPLE

This clustering violates both

principles:

Points in the same cluster are far

apart

Points in different cluster are close

together

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 B ETTER CLUSTERING
EXAMPLE

This clustering appears sensible.

But we need to use an objective

m e t r i c to o p t i m i z e c l u s te r
assignment.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 C LUSTERING TECHNIQUES

Agglomerative: Start with every element in its own cluster, and

iteratively join clusters together

Divisive: Start with one cluster and iteratively divide it into smaller
clusters

Hierarchical: Organize elements into a tree, leaves represent genes

and the length of the paths between leaves represents the distances
between genes. Similar genes lie within the same subtrees

Generally, finding the exact solution to a clustering problem is NP

hard.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K- MEANS CLUSTERING
A technique to partition a set of N points into K clusters

Each cluster is represented with a mean (a centroid) – hence ‘K-

means’

Input: A set V with N points (v1, v2 ... vn), the desired number of
clusters K and a distance measure between any two points d(v,w)

Output: A set X of K cluster centers that minimize the squared

error distortion D(V,X) over all possible choices of X.

�N
1
D(V, X) = min d (vi , xk )
2
N i=1 k

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K- MEANS CLUSTERING
For K=1, the problem becomes trivial: the centroid of all points is
the solution for Euclidean distances.
1 �
x= vi
N i
For K≥2 the problem becomes NP-complete

An efficient heuristic exists

Lloyd’s algorithm.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 L LOYD ’ S A LGORITHM

1. Arbitrarily assign the K cluster centers (this can significantly

influence the outcome)

2. while cluster centers keep changing

A. Compute the distance from each data point to the current cluster center Ci
(1 ≤ i ≤ K) and assign the point to the nearest cluster

B. After the assignment of all data points, compute new centers for each
cluster by taking the centroid of all the points in that cluster

3. Output cluster centers and assignments

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 1

2.00

Center
22 2
2
2 2 2
y

1 1 1
1 1
1
1.00 Center 1
1 11 1
1
1
1
1

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 2

2.00

22
1 Center 2
2 2 2
y

1 1 2
1 1
1
1.00 Center 11
1 11
1
1
1
1

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 3

2.00

11
1
Center
2 2 2 2
y

1 1 2
1 1
1
1.00 Center 1 1
1 11
2
2
1
2

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 4

2.00

11
1
2 2 2
y

1 1 2
1 1
1 Center 1
1.00 Center 2
1 11 2
2
2
2
2

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 5

2.00

11
1
2 2 2
y

1 1 2
1 1 1
Center
1
1.00 Center 2
1 12 2
2
2
2
2

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE
STEP 6

2.00

11
1
2 2 2
y

1 1
Center 2
1 1 1
1
1.00 Center 2
1 22 2
2
2
2
2

0.00
0.00 1.00 2.00

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-MEANS EXECUTION EXAMPLE

K=3 (different starting

K=2 K=3
points)

2.00 2.00 2.00

11 22 22
1 2 2
2 2 2 3 3 3 3 3 33
y

y
1 Center
1 1 2 Center
2 2 2 Center
2 2 Center
1 1 2 2 2 3 2 2 3
1 2 2
1.00 1.00 Center 3 1.00
2 Center 2
1 22 2 11 1 2 11 1
2 Center 1 3
2 3 Center111
2 1 1
2 3 1

0.00 0.00 0.00

0.00 1.00 2.00 0.00 1.00 2.00 0.00 1.00 2.00

x x x

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 H OW TO CHOOSE K ?
The simplest approach is to start with K=1 and increase K until the
squared error distortion (SED) stops decreasing

The problems is that K=N always achieves the value of 0 (each point is a
cluster), so we always keep increasing K.

Generally, need to add further constraints (e.g. model complexity) to obtain

non-trivial results
15

11.25
SED

7.5

3.75

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
K
CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]
 C ONSERVATIVE K-M EANS
A LGORITHM

Lloyd algorithm is fast but in each iteration it moves many data

points, not necessarily causing better convergence.

A more conservative method would be to move one point at a time

only if it improves the overall clustering cost

The smaller the clustering cost of a partition of data points is the better
that clustering is

Different methods (e.g. the squared error distortion) can be used to

measure this clustering cost

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 K-M EANS “G REEDY ” A LGORITHM
ProgressiveGreedyK-Means(k)
Select an arbitrary partition P into k clusters
while forever
bestChange ← 0
for every cluster C
for every element i not in C
if cost(P) – cost(Pi→C) > bestChange
bestChange ← cost(P) – cost(Pi→C)
i* ← I
C* ←C
if bestChange > 0
Change partition P by moving i* to C*
else
return P

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 CONCLUSION: LLOYD’S IN MORE EFFICIENT,
BOTH IN RUN-TIME AND IN BEST FOUND SED

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 HTTP://WWW.SPRINGERLINK.COM/CONTENT/K474381227655563/

Euclidean distance is not necessarily the best measure for co-

expression.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 H IERARCHICAL CLUSTERING
Instead of grouping into discrete clusters, produces a ‘classification’
tree, also called a dendrogram

A more intuitive example is probably obtained from molecular

sequence data (an early example of clustering applications)

We have a collection of aligned nucleotide sequences from different

species, and wish to construct their evolutionary hierarchy/history –
a phylogeny.

HTTP://WWW.SCIENCEMAG.ORG/CGI/REPRINT/310/5750/979.PDF

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 H IERARCHICAL CLUSTERING
Consider the following distance matrix on 5 nucleotide (partial
mitochondrial genome) sequences. The values are p-distances defined
as the number of nucleotide differences normalized by the length of
the sequence.
Human Chimpanzee Gorilla Orangutan Gibbon

Human - 0.0882682 0.102793 0.159598 0.179688

Chimpanzee - - 0.106145 0.170759 0.1875

Gorilla - - - 0.166295 0.1875

Orangutan - - - - 0.188616

Gibbon - - - - -

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 C LUSTERING PROCEDURE
At each step, we select the two closest sequences and join them to form a clade.

We then replace the two just joined sequences with their ancestor

This reduces the size of the data matrix by one

We need to compute the distances from the new ancestor to the remaining sequences
Human Chimpanzee Gorilla Orangutan Gibbon
Human - 0.0882682 0.102793 0.159598 0.179688
Chimpanzee - - 0.106145 0.170759 0.1875
Gorilla - - - 0.166295 0.1875
Orangutan - - - - 0.188616
Gibbon - - - - -

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 U PDATING DISTANCES
There are multiple strategies for computing the distances to the new
‘ancestral’ sequence a that joins sequences m and n

Single Linkage d(x, a) = min [d(x, m), d(x, n)]

Complete Linkage d(x, a) = max [d(x, m), d(x, n)]
UPGMA Unweighted Pair Group Method with d(x, m) + d(x, n)
d(x, a) =
Arithmetic Mean
2
WPGMA Weighted Pair Group Method with s(m)d(x, m) + s(n)d(x, n)
d(x, a) =
Arithmetic Mean
s(m) + s(n)
s(n) counts the number of actual
sequences represented by node n.

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 E XAMPLE CONTINUED
Use complete linkage. Joining human and chimp...
Human Chimpanzee Gorilla Orangutan Gibbon
Human - 0.0882682 0.102793 0.159598 0.179688
Chimpanzee - - 0.106145 0.170759 0.1875
Gorilla - - - 0.166295 0.1875
Orangutan - - - - 0.188616
Gibbon - - - - -

Human-Chimpanzee Gorilla Orangutan Gibbon

Human-Chimpanzee - 0.106145 0.170759 0.1875

Gorilla - - 0.166295 0.1875

Orangutan - - - 0.188616

Gibbon - - - -

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Human-Chimpanzee Gorilla Orangutan Gibbon
Human-Chimpanzee - 0.106145 0.170759 0.1875
Gorilla - - 0.166295 0.1875
Orangutan - - - 0.188616
Gibbon - - - -

Human-Chimpanzee-Gorilla Orangutan Gibbon

Human-Chimpanzee-Gorilla - 0.170759 0.1875
Orangutan - - 0.188616
Gibbon - - -

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Orangutan Gibbon Gibbon
Human-Chimpanzee-Gorilla 0.170759 0.1875
Hum-Chimp-Gor-Orang 0.188616
Orangutan - 0.188616
Gibbon - - Gibbon -

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 A NOTE ON WPGMA
Gorilla Orangutan Gibbon
Human-Chimpanzee 0.104469 0.165179 0.183594
Gorilla - 0.166295 0.1875
Orangutan - - 0.188616
Gibbon - - -

Orangutan Gibbon
Human-Chimpanzee-Gorilla 0.165551 0.184896
Orangutan - 0.188616
Gibbon - -

d(HCG-Orang) = 1/3 [2 d(HC-Orang) + d (Gor-Orang)]

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 B ACK TO MICROARRAYS ...
Clustering plots can be interpreted as gene/condition hierarchy

HTTP://UPLOAD.WIKIMEDIA.ORG/WIKIPEDIA/COMMONS/4/48/HEATMAP.PNG

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 A FEW OTHER APPLICATIONS

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Use clustering of similar sequences in protein databases to reduce
complexity and speed up comparisons. Each cluster of similar
sequences is represented by a single sequence.

Complexity reduction is an important application of clustering

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 The structure of proteins interactions can be represented by a graph

Node = proteins, Edges = interactions

Look for clusters (densely connected components) in graphs

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 Hierarchical clustering to
improve protein structure
prediction by merging the
predictions made by a large
number of alternative
conformation models

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 F URTHER READING ...

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]

 DEFINES THE
CONCEPT OF
‘ELEMENT
BELONGS TO A
PARTITION WITH
A PROBABILITY’

BUILD A MINIMUM
SPANNING TREE
AND DELETE
LONGEST EDGES
TO CREATE
PARTITIONS

CSE/BIMM/BENG 181 MAY 24, 2011 SERGEI L KOSAKOVSKY POND [[email protected]]