Ch.

16

What is clustering?
 Clustering: the process of grouping a set of objects
into classes of similar objects
 Documents within a cluster should be similar.
 Documents from different clusters should be
dissimilar.
 The commonest form of unsupervised learning
 Unsupervised learning = learning from raw data, as
opposed to supervised data where a classification of
examples is given
 A common and important task that finds many
applications in IR and other places
 Ch. 16

A data set with clear cluster structure

 How would
you design
an algorithm
for finding
the three
clusters in
this case?
 Sec. 16.2

Issues for clustering

 Representation for clustering
 Document representation
 Vector space? Normalization?
 Centroids aren’t length normalized
 Need a notion of similarity/distance
 How many clusters?
 Fixed a priori?
 Completely data driven?
 Avoid “trivial” clusters - too large or small
 If a cluster's too large, then for navigation purposes you've
wasted an extra user click without whittling down the set of
documents much.
Notion of similarity/distance
 Ideal: semantic similarity.
 Practical: term-statistical similarity
 We will use cosine similarity.
 Docs as vectors.
 For many algorithms, easier to think in
terms of a distance (rather than similarity)
between docs.
 We will mostly speak of Euclidean distance
 But real implementations use cosine similarity
Clustering Algorithms
 Flat algorithms
 Usually start with a random (partial) partitioning
 Refine it iteratively
 K means clustering
 (Model based clustering)
 Hierarchical algorithms
 Bottom-up, agglomerative
 (Top-down, divisive)
 Partitioning Algorithms
 Partitioning method: Construct a partition of n
documents into a set of K clusters
 Given: a set of documents and the number K
 Find: a partition of K clusters that optimizes the
chosen partitioning criterion
 Globally optimal
 Intractable for many objective functions
 Ergo, exhaustively enumerate all partitions
 Effective heuristic methods: K-means and K-
medoids algorithms
See also Kleinberg NIPS 2002 – impossibility for natural clustering
 Sec. 16.4

K-Means
 Assumes documents are real-valued vectors.
 Clusters based on centroids (aka the center of gravity
or mean) of points in a cluster, c:
 1 
μ(c)  
| c | xc
x

 Reassignment of instances to clusters is based on

distance to the current cluster centroids.
 (Or one can equivalently phrase it in terms of similarities)
 Sec. 16.4

K-Means Algorithm
Select K random docs {s1, s2,… sK} as seeds.
Until clustering converges (or other stopping criterion):
For each doc di:
Assign di to the cluster cj such that dist(xi, sj) is minimal.
(Next, update the seeds to the centroid of each cluster)
For each cluster cj
sj = (cj)
 Sec. 16.4

K Means Example
(K=2)
Pick seeds
Reassign clusters
Compute centroids
Reassign clusters
x x Compute centroids
x
x
Reassign clusters
Converged!
 Sec. 16.4

Termination conditions
 Several possibilities, e.g.,
 A fixed number of iterations.
 Doc partition unchanged.
 Centroid positions don’t change.

Does this mean that the docs in a

cluster are unchanged?
 Sec. 16.4

Convergence
 Why should the K-means algorithm ever reach a
fixed point?
 A state in which clusters don’t change.
 K-means is a special case of a general procedure
known as the Expectation Maximization (EM)
algorithm.
 EM is known to converge.
 Number of iterations could be large.
 But in practice usually isn’t
 Sec. 16.4

Convergence of K-Means
 Define goodness measure of cluster k as sum of
squared distances from cluster centroid:
 Gk = Σi (di – ck)2 (sum over all di in cluster k)
 G = Σk Gk
 Reassignment monotonically decreases G since
each vector is assigned to the closest centroid.
 Sec. 16.4

Convergence of K-Means
 Recomputation monotonically decreases each Gk
since (mk is number of members in cluster k):
 Σ (di – a)2 reaches minimum for:
 Σ –2(di – a) = 0
 Σ di = Σ a
 mK a = Σ di
 a = (1/ mk) Σ di = ck
 K-means typically converges quickly
 Sec. 16.4

Time Complexity
 Computing distance between two docs is O(M)
where M is the dimensionality of the vectors.
 Reassigning clusters: O(KN) distance computations,
or O(KNM).
 Computing centroids: Each doc gets added once to
some centroid: O(NM).
 Assume these two steps are each done once for I
iterations: O(IKNM).
 Sec. 16.4

Seed Choice
 Results can vary based on Example showing
random seed selection. sensitivity to seeds

 Some seeds can result in poor

convergence rate, or
convergence to sub-optimal
In the above, if you start
clusterings. with B and E as centroids
 Select good seeds using a heuristic you converge to {A,B,C}
(e.g., doc least similar to any and {D,E,F}
If you start with D and F
existing mean) you converge to
 Try out multiple starting points {A,B,D,E} {C,F}
 Initialize with the results of another
method.
 Sec. 16.4

K-means issues, variations, etc.

 Recomputing the centroid after every assignment
(rather than after all points are re-assigned) can
improve speed of convergence of K-means
 Assumes clusters are spherical in vector space
 Sensitive to coordinate changes, weighting etc.
 Disjoint and exhaustive
 Doesn’t have a notion of “outliers” by default
 But can add outlier filtering

Dhillon et al. ICDM 2002 – variation to fix some issues with small
document clusters
How Many Clusters?
 Number of clusters K is given
 Partition n docs into predetermined number of clusters
 Finding the “right” number of clusters is part of the
problem
 Given docs, partition into an “appropriate” number of
subsets.
 E.g., for query results - ideal value of K not known up front
- though UI may impose limits.
 Can usually take an algorithm for one flavor and
convert to the other.
K not specified in advance
 Say, the results of a query.
 Solve an optimization problem: penalize having
lots of clusters
 application dependent, e.g., compressed summary
of search results list.
 Tradeoff between having more clusters (better
focus within each cluster) and having too many
clusters
 K not specified in advance
 Given a clustering, define the Benefit for a
doc to be the cosine similarity to its
centroid
 Define the Total Benefit to be the sum of
the individual doc Benefits.

Why is there always a clustering of Total Benefit n?

 Ch. 17

Hierarchical Clustering
 Build a tree-based hierarchical taxonomy
(dendrogram) from a set of documents.
animal

vertebrate invertebrate

fish reptile amphib. mammal worm insect crustacean

 One approach: recursive application of a

partitional clustering algorithm.
Dendrogram: Hierarchical Clustering
 Clustering obtained
by cutting the
dendrogram at a
desired level: each
connected
component forms a
cluster.

21
 Sec. 17.1
Hierarchical Agglomerative Clustering
(HAC)
 Starts with each doc in a separate cluster
 then repeatedly joins the closest pair of
clusters, until there is only one cluster.
 The history of merging forms a binary tree
or hierarchy.

Note: the resulting clusters are still “hard” and induce a partition
 Sec. 17.2

Closest pair of clusters

 Many variants to defining closest pair of clusters
 Single-link
 Similarity of the most cosine-similar (single-link)
 Complete-link
 Similarity of the “furthest” points, the least cosine-similar
 Centroid
 Clusters whose centroids (centers of gravity) are the most
cosine-similar
 Average-link
 Average cosine between pairs of elements
 Sec. 17.2

Single Link Agglomerative Clustering

 Use maximum similarity of pairs:

sim (ci ,c j )  max sim ( x, y )

xci , yc j
 Can result in “straggly” (long and thin) clusters
due to chaining effect.
 After merging ci and cj, the similarity of the
resulting cluster to another cluster, ck, is:

sim ((ci  c j ), ck )  max( sim (ci , ck ), sim (c j , ck ))

 Sec. 17.2

Single Link Example

 Sec. 17.2

Complete Link
 Use minimum similarity of pairs:

sim(ci ,c j )  min sim( x, y)

xci , yc j
 Makes “tighter,” spherical clusters that are typically
preferable.
 After merging ci and cj, the similarity of the resulting
cluster to another cluster, ck, is:
sim ((ci  c j ), ck )  min( sim (ci , ck ), sim (c j , ck ))

Ci Cj Ck
 Sec. 17.2

Complete Link Example

 Sec. 17.3

Group Average
 Similarity of two clusters = average similarity of all pairs
within merged cluster.
1  
sim (ci , c j )    sim ( x , y )
ci  c j ( ci  c j  1) x( ci c j ) y( ci c j ): y  x
 Compromise between single and complete link.
 Two options:
 Averaged across all ordered pairs in the merged cluster
 Averaged over all pairs between the two original clusters
 No clear difference in efficacy
 Sec. 17.3

Computing Group Average Similarity

 Always maintain sum of vectors in each cluster.
 
s (c j )   x

xc j
 Compute similarity of clusters in constant time:

   
( s (ci )  s (c j ))  ( s (ci )  s (c j ))  (| ci |  | c j |)
sim (ci , c j ) 
(| ci |  | c j |)(| ci |  | c j | 1)
 Sec. 16.3

What Is A Good Clustering?

 Internal criterion: A good clustering will produce
high quality clusters in which:
 the intra-class (that is, intra-cluster) similarity is
high
 the inter-class similarity is low
 The measured quality of a clustering depends on
both the document representation and the
similarity measure used
 Sec. 16.3

External criteria for clustering quality

 Quality measured by its ability to discover some
or all of the hidden patterns or latent classes in
gold standard data
 Assesses a clustering with respect to ground truth
… requires labeled data
 Assume documents with C gold standard classes,
while our clustering algorithms produce K clusters,
ω1, ω2, …, ωK with ni members.
 Sec. 16.3

External Evaluation of Cluster Quality

 Simple measure: purity, the ratio between the
dominant class in the cluster πi and the size of
cluster ωi
1
Purity(i )  max j (nij ) j  C
ni
 Biased because having n clusters maximizes
purity
 Others are entropy of classes in clusters (or
mutual information between classes and
clusters)