Cluster Analysis
What is Cluster Analysis?
Finding groups of objects such that the objects in a
group will be similar (or related) to one another and
different from (or unrelated to) the objects in other
groups
Inter-cluster
Intra-cluster distances are
distances are maximized
minimized
What is Cluster Analysis?
Cluster: a collection of data objects
Similar to one another within the same cluster
Dissimilar to the objects in other clusters
Cluster analysis
Grouping a set of data objects into clusters
Clustering is unsupervised classification: no predefined
classes
Clustering is used:
As a stand-alone tool to get insight into data distribution
Visualization of clusters may unveil important information
As a preprocessing step for other algorithms
Efficient indexing or compression often relies on clustering
Some Applications of
Clustering
Pattern Recognition
Image Processing
cluster images based on their visual content
Bio-informatics
WWW and IR
document classification
cluster Weblog data to discover groups of similar access patterns
What Is Good Clustering?
A good clustering method will produce high quality
clusters with
high intra-class similarity
low inter-class similarity
The quality of a clustering result depends on both the
similarity measure used by the method and its
implementation.
The quality of a clustering method is also measured by
its ability to discover some or all of the hidden patterns.
Requirements of Clustering in
Data Mining
Scalability
Ability to deal with different types of attributes
Discovery of clusters with arbitrary shape
Minimal requirements for domain knowledge to
determine input parameters
Able to deal with noise and outliers
Insensitive to order of input records
High dimensionality
Incorporation of user-specified constraints
usability
Outliers
Outliers are objects that do not belong to any cluster
or form clusters of very small cardinality
cluster
outliers
In some applications we are interested in discovering
outliers, not clusters (outlier analysis)
Data Structures
attributes/dimensions
data matrix
x 11 ... x
1f
... x
1p
tuples/objects
(two modes)
... ... ... ... ...
x ... x ... x
i1 if ip
... ... ... ... ...
x ... x ... x
n1 nf np
objects
dissimilarity or distance
0
matrix d(2,1) 0
objects
(one mode) d(3,1 ) d ( 3, 2) 0
Assuming simmetric distance : : :
d(i,j) = d(j, i) d ( n , 1 ) d (n ,2) ... ... 0
Measuring Similarity in
Clustering
Dissimilarity/Similarity metric:
The dissimilarity d(i, j) between two objects i and j is expressed in
terms of a distance function, which is typically a metric:
d(i, j)0 (non-negativity)
d(i, i)=0 (isolation)
d(i, j)= d(j, i) (symmetry)
d(i, j) ≤ d(i, h)+d(h, j) (triangular inequality)
The definitions of distance functions are usually different
for interval-scaled, boolean, categorical, ordinal and ratio-
scaled variables.
Weights may be associated with different variables based
on applications and data semantics.
Type of data in cluster
analysis
Interval-scaled variables
e.g., salary, height
Binary variables
e.g., gender (M/F), has_cancer(T/F)
Nominal (categorical) variables
e.g., religion (Christian, Muslim, Buddhist, Hindu, etc.)
Ordinal variables
e.g., military rank (soldier, sergeant, lutenant, captain, etc.)
Ratio-scaled variables
population growth (1,10,100,1000,...)
Variables of mixed types
multiple attributes with various types
Similarity and Dissimilarity Between
Objects
Distance metrics are normally used to measure the
similarity or dissimilarity between two data objects
The most popular conform to Minkowski distance:
1/ p
p p p
L p (i , j ) x | | x x | ... | x x |
| x
i1
j1 i2 j2 in jn
where i = (xi1 , x , …, xin ) and j = (xj1 , xj2 , …, xjn ) are two n-dimensional
i2
data objects, and p is a positive integer
If p = 1, L1 is the Manhattan (or city block) distance:
L ( i , j ) | x x | | x x | ... | x x |
1 i1 j1 i2 j2 in jn
Similarity and Dissimilarity
Between Objects (Cont.)
If p = 2, L2 is the Euclidean distance:
d ( i , j ) (| x x | x x ... | x x
2 2 2
| | | )
i1 j1 i2 j2 in jn
Properties
d(i,j) 0
d(i,i) =0
d(i,j) = d(j,i)
d(i,j) d(i,k) + d(k,j)
Also one can use weighted distance:
d (i , j ) ( w | x x | w x ... w n | x x
2 2 2
|x | | )
1 i1 j1 2 i2 j2 in jn
Binary Variables
A binary variable has two states: 0 absent, 1 present
A contingency table for binary data object j
1 0 sum
i= (0011101001)
1 a b a b
J=(1001100110)
object i 0 c d c d
sum a c b d p
Simple matching coefficient distance :
d (i , j ) b c
Jaccard coefficient distance : a b c d
d (i , j ) b c
a b c
Binary Variables
Another approach is to define the similarity of two
objects and not their distance.
In that case we have the following:
Simple matching coefficient similarity:
s( i , j ) a d
a b c d
Jaccard coefficient similarity:
s( i , j ) a
a b c
Note that: s(i,j) = 1 – d(i,j)
Dissimilarity between Binary
Variables
Example (Jaccard coefficient)
all attributes are asymmetric binary
1 denotes presence or positive test
0 denotes absence or negative test
0 1
d( jack , mary ) 0 . 33
2 0 1
1 1
d( jack , jim ) 0 . 67
1 1 1
12
d( jim , mary ) 0 . 75
1 1 2
A simpler definition
Each variable is mapped to a bitmap (binary vector)
Jack: 101000
Mary: 101010
Jim:110000
Simple match distance:
number of non - common bit positions
d (i , j )
total number of bits
Jaccard coefficient:
number of 1' s in i j
d (i , j ) 1
number of 1' s in i j
Variables of Mixed Types
A database may contain all the six types of variables
symmetric binary, asymmetric binary, nominal, ordinal, interval
and ratio-scaled.
One may use a weighted formula to combine their effects.
Major Clustering Approaches
Partitioning algorithms: Construct random partitions and then
iteratively refine them by some criterion
Hierarchical algorithms: Create a hierarchical decomposition of the
set of data (or objects) using some criterion
Density-based: based on connectivity and density functions
Partitioning Algorithms: Basic
Concept
Partitioning method: Construct a partition of a database D
of n objects into a set of k clusters
k-means (MacQueen’67): Each cluster is represented by the center
of the cluster
k-medoids or PAM (Partition around medoids) (Kaufman &
Rousseeuw’87): Each cluster is represented by one of the objects in
the cluster
K-means Clustering
Partitional clustering approach
Each cluster is associated with a centroid (center point)
Each point is assigned to the cluster with the closest
centroid
Number of clusters, K, must be specified
K-means Clustering
Algorithm k-Means Clustering Algorithm
Input: a database D, of m records, r1, ..., rm and a desired number of
clusters k
Output: set of k clusters that minimizes the squared error criterion
Begin
randomly choose k records as the centroids for the k clusters;
repeat
assign each record, ri, to a cluster such that the distance between ri
and the cluster centroid (mean) is the smallest among the k clusters;
recalculate the centroid (mean) for each cluster based on the records
assigned to the cluster;
until no change;
End;
K-means Clustering Example
Sample 2-dimensional records for clustering example
RID Age Years_of_service
1 30 5
2 50 25
3 50 15 C1
4 25 5
5 30 10
6 55 25 C2
Assume that the number of desired clusters k is 2.
Let the algorithm choose records with RID 3 for cluster C1 and RID 6
for cluster C2 as the initial cluster centroids
The remaining records will be assigned to one of those clusters
during the first iteration of the repeat loop
K-means Clustering Example
The Euclidean distance between record rj and rk in n-
dimensional space is calculated as:
rj and rk represent the records wanted to calculate the distance between t
rk indicate to the C1 and C2 in current example
Record distance from C1 distance from C2 it joins cluster
1 22.4 32.0 C1
rj 2 10.0 5.0 C2
4 25.5 36.6 C1
5 20.6 29.2 C1
K-means Clustering Example
Now,the new means (centroids) for the two clusters are computed. The
mean for a cluster, Ci, with n records of m dimensions is the vector:
In our example records ( 1,3,4,5) belong to C1
And (2,6) belong to C2
so C1(new) =(1/4(30+50+25+30) , 1/2(50+55)=(33.75, 8.75)
C2(new) =(1/4(5+15+5+10) , 1/2(25+25)=(52.5, 25)
K-means Clustering Example
A second iteration proceeds to get the distance of each record with new
centroids
In the following table: calculate the distance of each record from the new
C1 and C2, and assign each to the suitable cluster
Record distance from C1 distance from C2 it joins cluster
1
2
3
4
5
6
calculate new C1 and C2 ?
tip : C1 will be (28.3, 6.7) and C2 will be (51.7, 21.7)
K-means Clustering Example
Move to the next iteration and do as the previous slide
Stop if you get same results
K-means Clustering – Details
Initial centroids are often chosen randomly.
Clusters produced vary from one run to another.
The centroid is (typically) the mean of the points in the
cluster.
‘Closeness’ is measured by Euclidean distance, cosine
similarity, correlation, etc.
Most of the convergence happens in the first few iterations.
Often the stopping condition is changed to ‘Until relatively few
points change clusters’
Complexity is O( n * K * I * d )
n = number of points, K = number of clusters,
I = number of iterations, d = number of attributes
Evaluating K-means Clusters
The terminating condition is usually the squared-error criterion. For clusters
C1, ..., Ck with means m1, ..., mk, the error is defined as:
rj is a data point in cluster Ci and mi is the corresponds mean of
the cluster
Solutions to Initial Centroids
Problem
Multiple runs
Helps, but probability is not on your side
Sample and use hierarchical clustering to
determine initial centroids
Select more than k initial centroids and then
select among these initial centroids
Select most widely separated
