CHAPTER 4: Clustering

TODAY’S TOPIC: CLUSTERING

Document clustering
Motivations
Document representations
Success criteria
Clustering algorithms
Partitional
Hierarchical
 Ch. 16

WHAT IS CLUSTERING?
Clustering: the process of grouping a set of objects into classes of
similar objects
 Documents within a cluster should be similar.
 Documents from different clusters should be
dissimilar.
The commonest form of unsupervised learning
 Unsupervised learning = learning from raw data, as
opposed to supervised data where a classification of
examples is given
 A common and important task that finds many applications in IR and other places
 Ch. 16

A DATA SET WITH CLEAR CLUSTER STRUCTURE

How would you

design an
algorithm for
finding the three
clusters in this
case?
 Sec. 16.1

APPLICATIONS OF CLUSTERING IN IR

Whole corpus analysis/navigation

Better user interface: search without typing
For improving recall in search applications
Better search results (like pseudo RF)
For better navigation of search results
Effective “user recall” will be higher
For speeding up vector space retrieval
Cluster-based retrieval gives faster search
 YAHOO! HIERARCHY ISN’T CLUSTERING BUT IS THE
KIND OF OUTPUT
www.yahoo.com/Science
YOU WANT FROM CLUSTERING
… (30)

agriculture biology physics CS space

... ... ... ...

...
dairy
crops botany cell AI courses craft
magnetism
forestry agronomy evolution HCI missions
relativity
GOOGLE NEWS: AUTOMATIC CLUSTERING GIVES AN
EFFECTIVE NEWS PRESENTATION METAPHOR
 Sec. 16.1

SCATTER/GATHER: CUTTING, KARGER, AND PEDERSEN

FOR VISUALIZING A DOCUMENT COLLECTION AND ITS
THEMES

Wise et al, “Visualizing the non-visual” PNNL

ThemeScapes, Cartia
 [Mountain height = cluster size]
 Sec. 16.1

FOR IMPROVING SEARCH RECALL

Cluster hypothesis - Documents in the same cluster behave
similarly with respect to relevance to information needs
Therefore, to improve search recall:
 Cluster docs in corpus a priori
 When a query matches a doc D, also return other docs in the
cluster containing D
Hope if we do this: The query “car” will also return docs
containing automobile
 Because clustering grouped together docs containing car with
thoseWhy
containing
mightautomobile.
this happen?
yippy.com – grouping search results11
 Sec. 16.2

ISSUES FOR CLUSTERING

Representation for clustering
 Document representation
 Vector space? Normalization?
 Centroids aren’t length normalized
 Need a notion of similarity/distance

How many clusters?

 Fixed a priori?
 Completely data driven?
 Avoid “trivial” clusters - too large or small
 If a cluster's too large, then for navigation purposes you've wasted an extra user click without whittling down
the set of documents much.
NOTION OF SIMILARITY/DISTANCE

Ideal: semantic similarity.

Practical: term-statistical similarity
We will use cosine similarity.
Docs as vectors.
For many algorithms, easier to think in terms
of a distance (rather than similarity) between
docs.
We will mostly speak of Euclidean distance
 But real implementations use cosine similarity
CLUSTERING ALGORITHMS

Flat algorithms
Usually start with a random (partial) partitioning
Refine it iteratively
 K means clustering
 (Model based clustering)
Hierarchical algorithms
Bottom-up, agglomerative
(Top-down, divisive)
HARD VS. SOFT CLUSTERING
Hard clustering: Each document belongs to exactly one
cluster
 More common and easier to do
Soft clustering: A document can belong to more than one
cluster.
 Makes more sense for applications like creating browsable
hierarchies
 You may want to put a pair of sneakers in two clusters: (i)
sports apparel and (ii) shoes
 You can only do that with a soft clustering approach.
We won’t do soft clustering today. See IIR 16.5, 18
 PARTITIONING ALGORITHMS

Partitioning method: Construct a partition of n

documents into a set of K clusters
Given: a set of documents and the number K
Find: a partition of K clusters that optimizes
the chosen partitioning criterion
Globally optimal
 Intractable for many objective functions
 Ergo, exhaustively enumerate all partitions

Effective heuristic methods: K-means and K-

medoids algorithms
See also Kleinberg NIPS 2002 – impossibility for natural clustering
 Sec. 16.4

K-MEANS
Assumes documents are real-valued vectors.
Clusters based on centroids (aka the center of gravity or mean) of
points in a cluster, c:  1 
μ(c) =
|c|
 x

xc

Reassignment of instances to clusters is based on distance to the

current cluster centroids.
 (Or one can equivalently phrase it in terms of similarities)
 Sec. 16.4

K-MEANS ALGORITHM
Select K random docs {s1, s2,… sK} as seeds.
Until clustering converges (or other stopping criterion):
For each doc di:
Assign di to the cluster cj such that dist(xi, sj) is minimal.
(Next, update the seeds to the centroid of each cluster)
For each cluster cj
sj = (cj)
 Sec. 16.4

K MEANS EXAMPLE
(K=2) Pick seeds
Reassign clusters
Compute centroids
Reassign clusters
x x Compute centroids
x
x
Reassign clusters
Converged!
 Sec. 16.4

TERMINATION CONDITIONS

Several possibilities, e.g.,

A fixed number of iterations.
Doc partition unchanged.
Centroid positions don’t change.

Does this mean that the docs in a

cluster are unchanged?
 Sec. 16.4

CONVERGENCE

Why should the K-means algorithm ever reach

a fixed point?
A state in which clusters don’t change.
K-means is a special case of a general
procedure known as the Expectation
Maximization (EM) algorithm.
EM is known to converge.
Number of iterations could be large.
 But in practice usually isn’t
 Sec. 16.4
Lower case!

CONVERGENCE OF K-MEANS

Define goodness measure of cluster k as sum

of squared distances from cluster centroid:
Gk = Σi (di – ck)2 (sum over all di in cluster
k)
G = Σk Gk
Reassignment monotonically decreases G since
each vector is assigned to the closest centroid.
 Sec. 16.4

CONVERGENCE OF K-MEANS
Recomputation monotonically decreases each Gk
since (mk is number of members in cluster k):
Σ (di – a)2 reaches minimum for:
Σ –2(di – a) = 0
Σ di = Σ a
mK a = Σ di
a = (1/ mk) Σ di = ck
K-means typically converges quickly
 Sec. 16.4

TIME COMPLEXITY
Computing distance between two docs is O(M) where M is the
dimensionality of the vectors.
Reassigning clusters: O(KN) distance computations, or O(KNM).
Computing centroids: Each doc gets added once to some centroid:
O(NM).
Assume these two steps are each done once for I iterations: O(IKNM).
 Sec. 16.4

SEED CHOICE
Results can vary based on random seed selection. Example showing
Some seeds can result in poor convergence rate, or sensitivity to seeds
convergence to sub-optimal clusterings.
 Select good seeds using a heuristic (e.g., doc least similar to
any existing mean)
 Try out multiple starting points
 Initialize with the results of another method. In the above, if you start
with B and E as centroids
you converge to {A,B,C}
and {D,E,F}
If you start with D and F
you converge to
{A,B,D,E} {C,F}
 Sec. 16.4

K-MEANS ISSUES, VARIATIONS, ETC.

Recomputing the centroid after every assignment (rather than after all
points are re-assigned) can improve speed of convergence of K-
means
Assumes clusters are spherical in vector space
 Sensitive to coordinate changes, weighting etc.
Disjoint and exhaustive
 Doesn’t have a notion of “outliers” by default
 But can add outlier filtering
Dhillon et al. ICDM 2002 – variation to fix some issues with small
document clusters
HOW MANY CLUSTERS?
Number of clusters K is given
 Partition n docs into predetermined number of clusters
Finding the “right” number of clusters is part of the problem
 Given docs, partition into an “appropriate” number of subsets.
 E.g., for query results - ideal value of K not known up front - though UI may impose
limits.

Can usually take an algorithm for one flavor and convert to the other.
K NOT SPECIFIED IN ADVANCE

Say, the results of a query.

Solve an optimization problem: penalize
having lots of clusters
application dependent, e.g., compressed
summary of search results list.
Tradeoff between having more clusters (better
focus within each cluster) and having too many
clusters
 K NOT SPECIFIED IN ADVANCE

Given a clustering, define the Benefit for

a doc to be the cosine similarity to its
centroid
Define the Total Benefit to be the sum of
the individual doc Benefits.

Why is there always a clustering of Total Benefit n?

PENALIZE LOTS OF CLUSTERS
For each cluster, we have a Cost C.
Thus for a clustering with K clusters, the Total Cost is KC.
Define the Value of a clustering to be =
Total Benefit - Total Cost.

Find the clustering of highest value, over all choices of K.

 Total benefit increases with increasing K. But can stop when it doesn’t increase by
“much”. The Cost term enforces this.
 Ch. 17

HIERARCHICAL CLUSTERING
Build a tree-based hierarchical taxonomy
(dendrogram) from a set of documents.
animal

vertebrate invertebrate

fish reptile amphib. mammal worm insect crustacean

One approach: recursive application of a

partitional clustering algorithm.
DENDROGRAM: HIERARCHICAL
CLUSTERING

 Clustering
obtained by
cutting the
dendrogram at a
desired level:
each connected
component
forms a cluster.

32
 Sec. 17.1

HIERARCHICAL AGGLOMERATIVE
CLUSTERING (HAC)
Starts with each doc in a separate
cluster
then repeatedly joins the closest pair of
clusters, until there is only one cluster.
The history of merging forms a binary
tree or hierarchy.
Note: the resulting clusters are still “hard” and induce a partition
 Sec. 17.2

CLOSEST PAIR OF CLUSTERS

Many variants to defining closest pair of clusters
Single-link
 Similarity of the most cosine-similar (single-link)

Complete-link
 Similarity of the “furthest” points, the least cosine-similar

Centroid
 Clusters whose centroids (centers of gravity) are the most cosine-similar

Average-link
 Average cosine between pairs of elements
 Sec. 17.2

SINGLE LINK AGGLOMERATIVE

CLUSTERING
Use maximum similarity of pairs:
sim(ci ,c j ) = max sim( x, y )
xci , yc j

Can result in “straggly” (long and thin) clusters

due to chaining effect.
After merging ci and cj, the similarity of the
resulting cluster to another cluster, ck, is:
sim((ci  c j ), ck ) = max(sim(ci , ck ), sim(c j , ck ))
 Sec. 17.2

SINGLE LINK EXAMPLE

 Sec. 17.2

COMPLETE LINK

sim(ci ,c j ) = min sim( x, y)

Use minimum similarity of pairs:

xci , yc j

Makes “tighter,” spherical clusters that are typically preferable.

After merging ci and cj, the similarity of the resulting cluster to another
cluster, ck, is:

sim((ci  c j ), ck ) = min(sim(ci , ck ), sim(c j , ck ))

Ci Cj Ck
 Sec. 17.2

COMPLETE LINK EXAMPLE

 Sec. 17.2.1

COMPUTATIONAL COMPLEXITY
In the first iteration, all HAC methods need to compute similarity of all
pairs of N initial instances, which is O(N2).
In each of the subsequent N−2 merging iterations, compute the
distance between the most recently created cluster and all other
existing clusters.
In order to maintain an overall O(N2) performance, computing
similarity to each other cluster must be done in constant time.
 Often O(N3) if done naively or O(N2 log N) if done more cleverly
 Sec. 17.3

GROUP AVERAGE

Similarity of two clusters = average similarity of all

1  
pairs within merged cluster.
sim(ci , c j ) =
ci  c j ( ci  c j − 1) 
 
 sim( x, y)
 
x( ci c j ) y( ci c j ): y  x

Compromise between single and complete link.

Two options:
 Averaged across all ordered pairs in the merged cluster
 Averaged over all pairs between the two original clusters

No clear difference in efficacy

 Sec. 17.3

COMPUTING GROUP AVERAGE SIMILARITY

Always maintain sum ofvectors in each cluster.


s (c j ) = x

xc j

Compute similarity of clusters in constant time:

   
( s (ci ) + s (c j )) • ( s (ci ) + s (c j )) − (| ci | + | c j |)
sim(ci , c j ) =
(| ci | + | c j |)(| ci | + | c j | −1)
 Sec. 16.3

WHAT IS A GOOD CLUSTERING?

Internal criterion: A good clustering will

produce high quality clusters in which:
the intra-class (that is, intra-cluster) similarity is
high
the inter-class similarity is low
The measured quality of a clustering depends on
both the document representation and the
similarity measure used
 Sec. 16.3

EXTERNAL CRITERIA FOR CLUSTERING QUALITY

Quality measured by its ability to discover

some or all of the hidden patterns or latent
classes in gold standard data
Assesses a clustering with respect to ground
truth … requires labeled data
Assume documents with C gold standard
classes, while our clustering algorithms produce
K clusters, ω1, ω2, …, ωK with ni members.
 Sec. 16.3

EXTERNAL EVALUATION OF CLUSTER QUALITY

Simple measure: purity, the ratio between the
dominant class in the cluster πi and the size of
cluster ωi
1
Purity (i ) = max j (nij ) j  C
ni

Biased because having n clusters maximizes

purity
Others are entropy of classes in clusters (or
mutual information between classes and clusters)
 Sec. 16.3

PURITY EXAMPLE
• • • • • •
• • • • • •
• • • • •

Cluster I Cluster II Cluster III

Cluster I: Purity = 1/6 (max(5, 1, 0)) = 5/6

Cluster II: Purity = 1/6 (max(1, 4, 1)) = 4/6

Cluster III: Purity = 1/5 (max(2, 0, 3)) = 3/5

 Sec. 16.3

RAND INDEX MEASURES BETWEEN PAIR DECISIONS.

HERE RI = 0.68
Different
Number of Same Cluster
Clusters in
points in clustering
clustering

Same class in
ground truth 20 24

Different
classes in 20 72
ground truth
 Sec. 16.3

RAND INDEX AND CLUSTER F-MEASURE

A+ D
RI =
A+ B +C + D
Compare with standard Precision and Recall:
A
P=
A
R=
A+ B

A+C
People also define and use a cluster F-
measure, which is probably a better measure.
FINAL WORD AND RESOURCES

In clustering, clusters are inferred from the data without

human input (unsupervised learning)
However, in practice, it’s a bit less clear: there are many
ways of influencing the outcome of clustering: number of
clusters, similarity measure, representation of documents,
...

Resources
 IIR 16 except 16.5
 IIR 17.1–17.3