UNIT-6

Cluster Analysis: Basic Concepts and Algorithms

Cluster Analysis: Basic Concepts and Algorithms: Overview: What
Is Cluster Analysis? Different Types of Clustering, Different Types of
Clusters; K-means: The Basic K-means Algorithm, K-means Additional
Issues, Bisecting K-means, Strengths and Weaknesses; Agglomerative
Hierarchical Clustering: Basic Agglomerative Hierarchical Clustering
Algorithm DBSCAN: Traditional Density Center-Based Approach, DBSCAN
Algorithm, Strengths and Weaknesses.

Cluster Analysis
Cluster analysis groups data objects based on information found only in the data
that describes the objects and their relationships. The goal is that the objects
within a group be similar (or related) to one another and different from (or
unrelated to) the objects in other groups. The greater the similarity (or
homogeneity) within a group and the greater the difference between groups, the
better or more distinct the clustering.
consider Figure which shows 20 points and three different ways of dividing them
into clusters. The shapes of the markers indicate cluster membership.

Figures (b) and (d) divide the data into two and six parts, respectively. However,
the apparent division of each of the two larger clusters into three sub clusters
may simply be an artifact of the human visual system.
Also, it may not be unreasonable to say that the points form four clusters, as
shown in Figure (c). This figure illustrates that the definition of a cluster is
imprecise and that the best definition depends on the nature of data and the
desired results. Cluster analysis is related to other techniques that are used to
divide data objects into groups.
 For instance, clustering can be regarded as a form of classification in that it
creates a labelling of objects with class (cluster) labels. However, it derives
these labels only from the data. In contrast, classification is supervised
classification; i.e., new, unlabelled objects are assigned a class label using a
model developed from objects with known class labels. For this reason,
cluster analysis is sometimes referred to as unsupervised classification. When
the term classification is used without any qualification within data mining, it
typically refers to supervised classification.
Also, while the terms segmentation and partitioning are sometimes used as
synonyms for clustering, these terms are frequently used for approaches outside
the traditional bounds of cluster analysis. For example, the term partitioning is
often used in connection with techniques that divide graphs into subgraphs and
that are not strongly connected to clustering. Segmentation often refers to the
division of data into groups using simple
techniques.
Example: An image can be split into segments based only on pixel intensity and
color, or people can be divided into groups based on their income.
Types of Clusterings:
Definition: An entire collection of clusters is commonly referred to as a
clustering.

Types of clusterings:
Hierarchical (nested) versus partitional (unnested)
Exclusive versus overlapping versus fuzzy
Complete versus partial.

Hierarchical versus Partitional:

Partitional Clustering: A partitional clustering is simply a division of the
set
of data objects into non-overlapping subsets (clusters) such that each data
object is in exactly one subset. Taken individually, each collection of
clusters in above Figures (b–d) is a partitional clustering.

Hierarchial Clustering: If we permit clusters to have sub clusters, then

we
obtain a hierarchical clustering, which is a set of nested clusters that are
organized as a tree. Each node (cluster) in the tree (except for the leaf nodes) is
the union of its children (sub clusters), and the root of the tree is the cluster
containing all the objects. Often, but not always, the leaves of the tree are
singleton clusters of individual data objects.
If we allow clusters to be nested, then one interpretation of Figure (a) is that it
has two sub clusters (b), each of which, in turn, has three sub clusters Figure (d).
The clusters shown in Figures (a–d), when taken in that order, also form a
hierarchical (nested) clustering with, respectively, 1, 2, 4, and 6 clusters on each
level.
Note: Finally, note that a hierarchical clustering can be viewed as a
sequence of
partitional clusterings and a partitional clustering can be obtained by taking any
member of that sequence; i.e., by cutting the hierarchical tree at a particular
level.

Exclusive versus Overlapping versus Fuzzy: Exclusive Clustering:

The clusterings shown in above Figure are all
exclusive, as they assign each object to a single cluster. There are many
situations in which a point could reasonably be placed in more than one cluster
and these situations are better addressed by non-exclusive clustering.
Overlapping Clustering: In the most general sense, an overlapping or

non-
exclusive clustering is used to reflect the fact that an object can simultaneously
belong to more than one group (class).For instance, a person at a university can
be both an enrolled student and an employee of the university.
A non-exclusive clustering is also often used when, for example, an object is
“between” two or more clusters and could reasonably be assigned to any of
these clusters. Imagine a point halfway between two of the clusters of Figure.
Rather than make a somewhat arbitrary assignment of the object to a single
cluster, it is placed in all of the “equally good” clusters.

Fuzzy clustering:
In a fuzzy clustering, every object belongs to every cluster with a membership
weight that is between 0 (absolutely doesn’t belong) and 1 (absolutely
belongs). In other words, clusters are treated as fuzzy sets. (Mathematically, a
fuzzy set is one in which an object belongs to every set with a weight that is
between 0 and 1. In fuzzy clustering, we often impose the additional constraint
that the sum of the weights for each object must equal 1.)
Similarly, probabilistic clustering techniques compute the probability with which
each point belongs to each cluster, and these probabilities must also sum to 1.
Because the membership weights or probabilities for any object sum to 1, a
fuzzy or probabilistic clustering does not address true multiclass situations, such
as the case of a student employee, where an object belongs to multiple classes.
Instead, these approaches are most appropriate for avoiding the arbitrariness of
assigning an object to only one cluster when it is close to several. In practice, a
fuzzy or probabilistic clustering is often converted to an exclusive clustering by
assigning each object to the cluster in which its membership weight or
probability is highest.
Complete versus Partial: Complete Clustering: A complete clustering
assigns every object to a cluster,
whereas a partial clustering does not.
partial clustering: Some objects in a data set may not belong to well

defined
groups. Many times objects in the data set represent noise, outliers, or
“uninteresting background.”

For example: some newspaper stories share a common theme, such as

global
warming, while other stories are more generic or one-of-a-kind. Thus, to find the
important topics in last month’s stories, we often want to search only for clusters
of documents that are tightly related by a common theme.
In other cases, a complete clustering of the objects is desired. For example, an
application that uses clustering to organize documents for browsing needs to
Types of
guarantee thatClusters:
all documents can be browsed.

Well-Separated Cluster:
A cluster is a set of objects in which each object is closer (or more similar) to
every other object in the cluster than to any object not in the cluster.
Sometimes a threshold is used to specify that all the objects in a cluster must
be sufficiently close (or similar) to one another. Figure(a) gives an example of
well separated clusters that consists of two groups of points in a two-
dimensional space. The distance between any two points in different groups is
larger than the distance between any two points within a group. Well-
separated clusters do not need to be globular, but can have any shape.

Prototype-Based Cluster: A cluster is a set of objects in which each object

is closer (more similar) to the prototype that defines the cluster than to the
prototype of any other cluster. For data with continuous attributes, the prototype
of a cluster is often a centroid, i.e., the average (mean) of all the points in the
cluster. When a centroid is not meaningful, such as when the data has
categorical attributes, the prototype is often a medoid, i.e., the most
representative point of a cluster. For many types of data, the prototype can be
regarded as the most central point, and in such instances, we commonly refer to
prototype based clusters as center-based clusters.

Graph-Based Cluster: If the data is represented as a graph, where the

nodes
are objects and the links represent connections among objects, then a cluster
can be defined as a connected component; i.e., a group of objects that are
connected to one another, but that have no connection to objects outside the
group. Example: Example of graph-based clusters is a contiguity-based cluster,
where
two objects are connected only if they are within a specified distance of each
other. This implies that each object in a contiguity-based cluster is closer to
some other object in the cluster than to any point in a different cluster.
Figure (c) shows an example of such clusters for two-dimensional points. This
definition of a cluster is useful when clusters are irregular or intertwined.
However, this approach can have trouble when noise is present since, as
illustrated by the two spherical clusters of Figure(c), a small bridge of points can
merge two distinct clusters.

Density-Based Cluster: A cluster is a dense region of objects that is

surrounded by a region of low density. Figure(d) shows some density-based
clusters for data created by adding noise to the data of Figure (c). The two
circular clusters are not merged, as in Figure(c), because the bridge between
them fades into the noise. Likewise, the curve that is present in Figure(c) also
fades into the noise and does not form a cluster in Figure(d).
A density based definition of a cluster is often employed when the clusters are
irregular or intertwined, and when noise and outliers are present. By contrast, a
contiguity based definition of a cluster would not work well for the data of
Figure(d) because the noise would tend to form bridges between clusters.

Shared-Property(Conceptual Clusters):
More generally, we can define a cluster as a set of objects that share some
property. This definition encompasses all the previous definitions of a cluster;
e.g., objects in a center based cluster share the property that they are all
closest to the same centroid or medoid. However, the shared-property
approach also includes new types of clusters.
Consider the clusters shown in Figure(e). A triangular area (cluster) is adjacent to
a rectangular one, and there are two intertwined circles (clusters). In both cases,
a clustering algorithm would need a very specific concept of a cluster to
successfully detect these clusters. The process of finding such clusters is called
conceptual clustering.

K-means:
Prototype-based clustering techniques create a one-level partitioning of the data
objects.
The Basic K-means Algorithm:
Procedure: We first choose K initial centroids, where K is a user specified
parameter, namely, the number of clusters desired. Each point is then assigned
to the closest centroid, and each collection of points assigned to a centroid is a
cluster. The centroid of each cluster is then updated based on the points
assigned to the cluster. We repeat the assignment and update steps until no
point changes clusters, or equivalently, until the centroids remain the same.

Time and Space Complexity:

The space requirements for K-means are modest because only the data points
and centroids are stored. Specifically, the storage required is O((m + K)n),
where m is the number of points and n is the number of attributes. The time
requirements for K-means are also modest—basically linear in the number of
data points. In particular, the time required is O(I×K×m×n), where I is the
number of iterations required for convergence. As mentioned, I is often small
and can usually be safely bounded, as most changes typically occur in the first
few iterations. Therefore, K-means is linear in m, the number of points, and is
efficient as well as simple provided that K, the number of clusters, is
significantly less than m.

Method:
1.Randomly assign K objects from the dataset(D) as cluster
centres(C)
2.(Re) Assign each object to which object is most similar based
upon mean values.
3.Update Cluster means, i.e., Recalculate the mean of each
cluster with the updated values.
4.Repeat Step 2 until no change occurs.
 Figure – K-mean
Clustering
Flowchart:

Figure
– K-mean Clustering

\
Hierarchial Clustering
What is Hierarchical Clustering?
Hierarchical clustering is a method of cluster analysis in data
mining that creates a hierarchical representation of the clusters
in a dataset. The method starts by treating each data point as a
separate cluster and then iteratively combines the closest
clusters until a stopping criterion is reached. The result of
hierarchical clustering is a tree-like structure, called a
dendrogram, which illustrates the hierarchical relationships
among the clusters

Types of Hierarchical Clustering

Basically, there are two types of hierarchical Clustering:
1.Agglomerative Clustering
2.Divisive clustering
1. Agglomerative Clustering
Initially consider every data point as an individual Cluster and
at every step, merge the nearest pairs of the cluster. (It is a
bottom-up method). At first, every dataset is considered an
individual entity or cluster. At every iteration, the clusters merge
with different clusters until one cluster is formed.
The algorithm for Agglomerative Hierarchical Clustering is:


 Calculate the similarity of one cluster with all the other
 clusters (calculate proximity matrix)
 Consider every data point as an individual cluster
 Merge the clusters which are highly similar or close to each
other.
Recalculate the proximity matrix for each cluster
Repeat Steps 3 and 4 until only a single cluster remains.
 Step-1: Consider each alphabet as a single cluster and
calculate the distance of one cluster from all the other
clusters.
 Step-2: In the second step comparable clusters are merged
together to form a single cluster. Let’s say cluster (B) and
cluster (C) are very similar to each other therefore we merge
them in the second step similarly to cluster (D) and (E) and at
last, we get the clusters [(A), (BC), (DE), (F)]

Step-3: We recalculate the proximity according to the
algorithm and merge the two nearest clusters([(DE), (F)])
together to form new clusters as [(A), (BC), (DEF)]
Step-4: Repeating the same process; The clusters DEF and

BC are comparable and merged together to form a new
cluster. We’re now left with clusters [(A), (BCDEF)].
Step-5: At last, the two remaining clusters are merged
 together to form a single cluster [(ABCDEF)].

Agglomerative Clustering Algorithm

Following are the steps in agglomerative clustering.
1.We start by assigning each data point to its own cluster.
2.Next, we compute the distance between each pair of clusters
and select the pair of clusters with the smallest distance.
3.Then, we merge the pair of clusters with the smallest distance
into a single cluster and update the distance between the
newly formed cluster and every other cluster.
4.We repeat steps 2 and 3 until all data points are in one cluster.

Calculation of Distance Between Two

Clusters
The distance between clusters in agglomerative clustering can be
calculated using three approaches namely single linkage, complete
linkage, and average linkage.

 In the single linkage approach, we take the distance between

the nearest points in two clusters as the distance between the
clusters.
  In the complete linkage approach, we take the distance
between the farthest points in two clusters as the distance
between the clusters.
 In the average linkage approach, we take the average distance
between each pair of points in two given clusters as the
distance between the clusters. You can also take the distance
between the centroids of the clusters as their distance from
each other.

A (1, 1), B(2, 3), C(3, 5), D(4,5), E(6,6), and F(7,5) and try to cluster
them.
To perform clustering, we will first create a distance matrix
consisting of the distance between each point in the dataset. The
distance matrix looks as follows.

Using Single Linkage approach

Step 1: First, we will consider each data point as a single cluster.
After this, we will start combining the clusters.

Step 2: To combine the individual clusters, we can consider the

following points.
1.Point A is closest to point B.
2.Point B is at a similar distance to points A and C.
3.Point C is closest to point D.
4.Point D is closest to point C.
5.Point E is closest to F.
6.Point F is closest to E.

From the above points, we can combine (C, D) and (E, F) as clusters.
We will name them CD and EF respectively. As we have ambiguity
for points A and B, let us not combine them and treat them as
individual clusters.
Step 3: Now, we will calculate the minimum distance between
clusters A, B, CD, and EF. You can observe that

1.Cluster A is closest to B.
2.Cluster B is closest to A as well as CD.
3.Cluster EF is closest to CD.

Using the above information let us combine B and CD and name the
cluster BCD.
Step 4: After this, The cluster BCD is at the same distance from A
and EF. Hence, we will first merge BCD and EF to form BCDEF.
Step 5: Finally, we will merge A and BCDEF to form the cluster
ABCDEF. Using the above steps, we will get the following
dendrogram.
In the above example, if we combine (A, B), (C, D), and (E, F)
together in step 2 above, we will get clusters AB, CD, and EF.
Now, The minimum distance of CD is the same from AB and EF. Let
us merge AB and CD to form ABCD first. Next, we will combine ABCD
and EF to obtain the cluster ABCDEF.

As a result, we will get the following dendrogram.

Instead of combining AB and CD in the previous example, let us first
combine CD and EF to form CDEF. Then, we can combine AB with
CDEF to form the cluster ABCDEF. As a result, we will get the
following dendrogram.

K-means: Additional Issues

Handling Empty Clusters:
One of the problems with the basic K-means algorithm is that empty clusters can
be obtained if no points are allocated to a cluster during the assignment step. If
this happens, then a strategy is needed to choose a replacement centroid, since
otherwise, the squared error will be larger than necessary. Approach 1:
One
approach is to choose the point that is farthest away from any current centroid. If
nothing else, this eliminates the point that currently contributes most to the total
squared error.

Approach 2: Another approach is to choose the replacement centroid

at
random from the cluster that has the highest SSE. This will typically split the
cluster and reduce the overall SSE of the clustering. If there are several empty
clusters, then this process can be repeated several times.
Outliers:
When the squared error criterion is used, outliers can unduly influence the
clusters that are found. In particular, when outliers are present, the
resulting cluster centroids (prototypes) are typically not as representative
as they otherwise would be and thus, the SSE will be higher. Because of
this, it is often
useful to discover outliers and eliminate them beforehand. It is important,
however, to appreciate that there are certain clustering applications for
which outliers should not be eliminated.
When clustering is used for data compression, every point must be clustered,
and in some cases, such as financial analysis, apparent outliers, e.g., unusually
profitable customers, can be the most interesting points. An obvious issue is how to
identify outliers. There are a number of techniques for
identifying outliers. If we use approaches that remove outliers before clustering,
we avoid clustering points that will not cluster well. Alternatively, outliers can also
be identified in a postprocessing step. For
instance, we can keep track of the SSE contributed by each point, and eliminate
those points with unusually high contributions, especially over multiple runs.
Also, we often want to eliminate small clusters because they frequently
represent groups of outliers.

Reducing the SSE with Postprocessing:

An obvious way to reduce the SSE is to find more clusters, i.e., to use a larger K.
In many cases, we would like to improve the SSE, but don’t want to increase the
number of clusters. This is often possible because K-means typically converges to
a local minimum.
Various techniques are used to “fix up” the resulting clusters in order to produce
a clustering that has lower SSE. The strategy is to focus on individual clusters
since the total SSE is simply the sum of the SSE contributed by each cluster. (We
will use the terms total SSE and cluster SSE, respectively, to avoid any potential
confusion.) We can change the total SSE by performing various operations on the
clusters, such as splitting or merging clusters.
One commonly used approach is to employ alternate cluster splitting and
merging phases. During a splitting phase, clusters are divided, while during a
merging phase, clusters are combined. In this way, it is often possible to escape
local SSE minima and still produce a clustering solution with the desired number
of clusters. The following are some techniques used in the splitting and merging
phases.

Two strategies that decrease the total SSE by increasing the

number of clusters are the following:
Split a cluster: The cluster with the largest SSE is usually chosen, but

we
could also split the cluster with the largest standard deviation for one particular
attribute.

Introduce a new cluster centroid: Often the point that is farthest from
any
cluster center is chosen. We can easily determine this if we keep track of the SSE
contributed by each point. Another approach is to choose randomly from all
points or from the points with the highest SSE with respect to their closest
centroids.
Two strategies that decrease the number of clusters, while trying
to minimize the increase in total SSE, are the following:
Disperse a cluster:by removing the centroid that
This is accomplished
corresponds to the cluster and reassigning the points to other clusters. Ideally,
the cluster that is dispersed should be the one that increases the total SSE the
least.

Merge two clusters: The clusters with the closest centroids are typically
chosen, although another, perhaps better, approach is to merge the two clusters
that result in the smallest increase in total SSE. These two merging strategies
are the same ones that are used in the hierarchical clustering techniques known
as the centroid method and Ward’s method, respectively.

Updating Centroids Incrementally:

Instead of updating cluster centroids after all points have been assigned to a
cluster, the centroids can be updated incrementally, after each assignment of
a point to a cluster. Notice that this requires either zero or two updates to
cluster centroids at each step, since a point either moves to a new cluster (two
updates) or stays in its current cluster (zero updates).
Using an incremental update strategy guarantees that empty clusters are not
produced because all clusters start with a single point, and if a cluster ever has
only one point, then that point will always be reassigned to the same cluster. In
addition, if incremental updating is used, the relative weight of the point
being added can be adjusted; e.g., the weight of points is often decreased as the
clustering proceeds. While this can result in better accuracy and faster
convergence, it can be difficult to make a good choice for the relative weight,
especially in a wide variety of situations. These update issues are similar to those
involved in updating weights for artificial neural networks. Yet another benefit of
incremental updates has to do with using objectives other than “minimize SSE.”
Suppose that we are given an arbitrary objective function to measure the
goodness of a set of clusters. When we process an individual point, we can
compute the value of the objective function for each possible cluster assignment,
and then choose the one that optimizes the objective.
On the negative side, updating centroids incrementally introduces an order
dependency. In other words, the clusters produced usually depend on the order
in which the points are processed. Although this can be addressed by
randomizing the order in which the points are processed, the basic K-means
approach of updating the centroids after all points have been assigned to
clusters has no order dependency. Also, incremental updates are slightly more
expensive. However, K-means converges rather quickly, and therefore, the
number of points switching clusters quickly becomes relatively small.
Advantages:
• Easy to implement.
 can be used for a wide variety of data types.

 It is also quite efficient, even though multiple runs are often performed.
•
With a large number of variables, K-Means may be computationally faster
than hierarchical clustering (if K is small).

• k-Means may produce tighter clusters than hierarchical clustering.

• An instance can change cluster (move to another cluster) when the centroids
are recomputed.
Disadvantages:
• Difficult to predict the number of clusters (K-Value)
• Initial seeds have a strong impact on the final results
• The order of the data has an impact on the final results
• Sensitive to scale: rescaling your datasets (normalization or standardization)
will completely change results. While this itself is not bad, not realizing that you
have to spend extra attention to scaling your data might be bad.

 K-means is not suitable for all types of data.

 K-means also has trouble clustering data that contains outliers. It cannot handle

 non-globular clusters or clusters of different sizes and

densities, although it can typically find pure sub clusters if a large enough
number of clusters is specified.

 k-means is restricted to data for which there is a notion of a center (centroid).

Bisecting K-means
Idea: The bisecting K-means algorithm is a straightforward extension of the
basic K-means algorithm that is based on a simple idea: to obtain K clusters, split
the set of all points into two clusters, select one of these clusters to split, and so
on, until K clusters have been produced.
There are a number of different ways to choose which cluster to split. We can
choose the largest cluster at each step, choose the one with the largest SSE, or
use a criterion based on both size and SSE. Different choices result in different
clusters. Because we are using the K-means algorithm “locally,” i.e., to bisect
individual clusters, the final set of clusters does not represent a clustering that is
a local minimum with respect to the total SSE. Thus, we often refine the resulting
clusters by using their cluster centroids as the initial centroids for the standard
K-means algorithm.
Algorithm:

K-means as an Optimization Problem:

Given an objective function such as “minimize SSE,” clustering can be treated as
an optimization problem. One way to solve this problem—to find a global
optimum—is to enumerate all possible ways of dividing the points into clusters
and then choose the set of clusters that best satisfies the objective function, e.g.,
that minimizes the total SSE.
This strategy is computationally infeasible and as a result, a more practical
approach is needed, even if such an approach finds solutions that are not
guaranteed to be optimal. One technique, which is known as gradient descent, is
based on picking an initial solution and then repeating the following two steps:
compute the change to the solution that best optimizes the objective function
and then update the solution. We assume that the data is one-dimensional, i.e.,
This does not change anything essential, but greatly

simplifies the notation.

Derivation of K-means as an Algorithm to Minimize the SSE:

The centroid for the K-means algorithm can be mathematically derived when the
proximity function is Euclidean distance and the objective is to minimize the SSE.
Specifically, we investigate how we can best update a cluster centroid so that the
cluster SSE is minimized.

Here, Ci is the ith cluster, x is a point in Ci, and ci is the mean of the ith cluster.
We can solve for the kth centroid ck, which minimizes Equation, by
differentiating the SSE, setting it equal to 0, and solving, as indicated below.
Thus, as previously indicated, the best centroid for minimizing the SSE of a
cluster is the mean of the points in the cluster.

Derivation of K-means for SAE:

To demonstrate that the K-means algorithm can be applied to a variety of
different objective functions, we consider how to partition the data into K
clusters such that the sum of the Manhattan (L1) distances of points from
the center of their clusters is minimized. We are seeking to minimize the
sum of the L1 absolute errors (SAE) as given by the following equation,
where distL1 is the L1 distance. Again, for notational simplicity, we use
one-dimensional data, i.e., distL1 = |ci −x|.

We can solve for the kth centroid ck, which minimizes Equation 7.5, by
differentiating the SAE, setting it equal to 0, and solving.
If we solve for ck, we find that ck = median{x ∈ Ck}, the median of the points in
the cluster. The median of a group of points is straightforward to compute and
less susceptible to distortion by outliers.

Agglomerative Hierarchical Clustering

There are two basic approaches for generating a hierarchical clustering:

Agglomerative: Start with the points as individual clusters and, at each

step,
merge the closest pair of clusters. This requires defining a notion of cluster
proximity.

Divisive: Start with one, all-inclusive cluster and, at each step, split a
cluster
until only singleton clusters of individual points remain. In this case, we need to
decide which cluster to split at each step and how to do the splitting.
Agglomerative hierarchical clustering techniques are by far the most common. A
hierarchical clustering is often displayed graphically using a tree-like diagram
called a dendrogram, which displays both the cluster-sub cluster relationships
and the order in which the clusters were merged (agglomerative view) or split
(divisive view). For sets of two-dimensional points, a hierarchical clustering can
also be
graphically represented using a nested cluster diagram. Figure shows an
example of these two types of figures for a set of four two-dimensional points.
Basic Agglomerative Hierarchical Clustering Algorithm:
Many agglomerative hierarchical clustering techniques are variations on a single
approach: starting with individual points as clusters, successively merge the two
closest clusters until only one cluster remains.

Time and Space Complexity

The basic agglomerative hierarchical clustering algorithm just presented uses a
proximity matrix. This requires the storage of proximities (assuming the
proximity matrix is symmetric) where m is the number of data points. The space
needed to keep track of the clusters is proportional to the number of clusters,
which is m−1, excluding singleton clusters. Hence, the total space complexity is

The analysis of the basic agglomerative hierarchical clustering algorithm is also

straightforward with respect to computational complexity. time is required
to compute the proximity matrix. After that step, there are m−1 iterations
involving steps 3 and 4 because there are m clusters at the start and two
clusters are merged during each iteration. If performed as a linear search of the
proximity matrix, then for the ith iteration, Step 3 requires time,
which is proportional to the current number of clusters squared. Step 4 requires
O(m−i + 1) time to update the proximity matrix after the merger of two clusters.
(A cluster merger affects O(m − i + 1) proximities for the techniques that we
consider.) Without modification, this would yield a time complexity of . If
the distances from each cluster to all other clusters are stored as a sorted list (or
heap), it is possible to reduce the cost of finding the two closest clusters to
O(m−i + 1). However, because of the additional complexity of keeping data in a
sorted list or heap, the overall time required for a hierarchical clustering based
on Algorithm is

Specific Techniques
Sample Data

To illustrate the behaviour of the various hierarchical clustering algorithms, we

will use sample data that consists of six two-dimensional points, which are
shown in Figure. The x and y coordinates of the points and the Euclidean
distances between them are shown in Tables , respectively.

Single Link or MIN

For the single link or MIN version of hierarchical clustering, the proximity of two
clusters is defined as the minimum of the distance (maximum of the similarity)
between any two points in the two different clusters. Using graph terminology, if
you start with all points as singleton clusters and add links between points one at
a time, shortest links first, then these single links combine the points into
clusters. The single link technique is good at handling non-elliptical shapes, but is
sensitive to noise and outliers.

Example (Single Link): Figure shows the result of applying the single

link
technique to our example data set of six points. Figure (a) shows the nested
clusters as a sequence of nested ellipses, where the numbers associated with the
ellipses indicate the order of the clustering. Figure(b) shows the same
information, but as a dendrogram. The height at which two clusters are merged
in the dendrogram reflects the distance of the two clusters. For instance, from
Table, we see that the distance between points 3 and 6 is 0.11, and that is the
height at which they are joined into one cluster in the dendrogram. As another
example, the distance between clusters{3,6}and {2,5} is given by dist({3,6},
{2,5}) = min(dist(3,2),dist(6,2),dist(3,5),dist(6,5)) = min(0 .15,0.25,0.28,0.39)
=0 .15.

Complete Link or MAX or CLIQUE

For the complete link or MAX version of hierarchical clustering, the proximity of
two clusters is defined as the maximum of the distance (minimum of the
similarity) between any two points in the two different clusters. Using graph
terminology, if you start with all points as singleton clusters and add links
between points one at a time, shortest links first, then a group of points is not a
cluster until all the points in it are completely linked, i.e., form a clique.
Complete link is less susceptible to noise and outliers, but it can break large
clusters and it favours globular shapes.

Example (Complete Link): Figure shows the results of applying MAX to

the
sample data set of six points. As with single link, points 3 and 6 are merged first.
However, {3,6} is merged with {4}, instead of {2,5} or {1} because
Group Average:
For the group average version of hierarchical clustering, the proximity of two
clusters is defined as the average pairwise proximity among all pairs of points
in the different clusters. This is an intermediate approach between the single
and complete link approaches. Thus, for group average, the cluster proximity
proximity(Ci, C j) of clusters Ci and Cj, which are of size mi and mj,
respectively, is expressed by the following equation:

Example (Group Average). Figure shows the results of applying the group
average approach to the sample data set of six points. To illustrate how group
average works, we calculate the distance between some clusters.
Because dist({3,6,4},{2,5}) is smaller than dist({3,6,4},{1}) and dist({2,5},
{1}), clusters {3,6,4} and {2,5} are merged at the fourth stage.

Ward’s Method :
For Ward’s method, the proximity between two clusters is defined as the
increase in the squared error that results when two clusters are merged. Thus,
this method uses the same objective function as K-means clustering. While it
might seem that this feature makes Ward’s method somewhat distinct from
other hierarchical techniques, it can be shown mathematically that Ward’s
method is very similar to the group average method when the proximity between
two points is taken to be the square of the distance between them.

Example (Ward’s Method): Figure shows the results of applying Ward’s

method to the sample data set of six points. The clustering that is produced is
different from those produced by single link, complete link, and group average.
Centroid method:
Centroid methods calculate the proximity between two clusters by calculating
the distance between the centroids of clusters. These techniques may seem
similar to K-means, but as we have remarked, Ward’s method is the correct
hierarchical analog. Centroid methods also have a characteristic—often
considered bad—that is not possessed by the other hierarchical clustering
techniques that we have discussed: the possibility of inversions. Specifically, two
clusters that are merged can be more similar (less distant) than the pair of
clusters that were merged in a previous step. For the other methods, the
distance between merged clusters monotonically increases (or is, at worst, non-
increasing) as we proceed from singleton clusters to one all-inclusive cluster.
Advantages:
• Hierarchical clustering outputs a hierarchy, i.e a structure that is more
informative than the unstructured set of flat clusters returned by k-means.
Therefore, it is easier to decide on the number of clusters by looking at the
dendrogram
. • Easy to implement.

Dis Advantages:

 Agglomerative hierarchical clustering algorithms are expensive in terms of

their computational and storage requirements.
All merges are final can also cause trouble for noisy, high-dimensional data,

such as document data.
It is not possible to undo the previous step: once the instances have been
 assigned to a cluster, they can no longer be moved around.
Time complexity: not suitable for large datasets.


  Initial seeds have a strong impact on the final results.

 The order of the data has an impact on the final results.

 Very sensitive to outliers.

DBSCAN
Density-based clustering locates regions of high density that are separated from
one another by regions of low density. DBSCAN is a simple and effective density-
based clustering algorithm that illustrates a number of important concepts that
are important for any density-based clustering approach. In this section, we focus
solely on DBSCAN after first considering the key notion of density.

Traditional Density: Center-Based Approach:

In the center-based approach, density is estimated for a particular point in the
data set by counting the number of points within a specified radius, Eps, of that
point. This includes the point itself. This technique is graphically illustrated by
Figure. The number of points within a radius of Eps of point A is 7, including A
itself.
This method is simple to implement, but the density of any point will depend on
the specified radius. For instance, if the radius is large enough, then all points
will have a density of m, the number of points in the data set. Likewise, if the
radius is too small, then all points will have a density of 1. An approach for
deciding on the appropriate radius for low-dimensional data is given in the next
section in the context of our discussion of DBSCAN.

Center based density

Classification of Points According to Center-Based Density:
The center-based approach to density allows us to classify a point as being (1) in
the interior of a dense region (a core point), (2) on the edge of a dense region (a
border point), or (3) in a sparsely occupied region (a noise or background point).
Figure graphically illustrates the concepts of core, border, and noise points using
a collection of two-dimensional points. The following text provides a more precise
description.

Core points: These points are in the interior of a density-based cluster. A

point
is a core point if there are at leastMinPts within a distance of Eps, where MinPts
and Eps are user-specified parameters. In Figure, point A is a core point for the
radius (Eps) if MinPts≥ 7.
Border points: A border point is not a core point, but falls within the
neighbourhood of a core point. In Figure, point B is a border point. A border point
can fall within the neighbourhoods of several core points.

Noise points: A noise point is any point that is neither a core point
nor a
border point. In Figure, point C is a noise point.
The DBSCAN Algorithm
Given the previous definitions of core points, border points, and noise
points, the DBSCAN algorithm can be informally described as follows.
Any two core points that are close enough—within a distance Eps of one
another—are put in the same cluster. Likewise, any border point that is
close enough to a core point is put in the same cluster as the core point.
(Ties need to be resolved if a border point is close to core points from
different clusters.) Noise points are discarded. The formal details are given
in Algorithm. This algorithm uses the same
concepts and finds the same clusters as the original DBSCAN, but is
optimized for simplicity, not efficiency.
Time and Space Complexity:
The basic time complexity of the DBSCAN algorithm is O(m × time to find points in
the Eps-neighbourhood), where m is the number of points. In the worst case,
this complexity is However, in low-dimensional spaces (especially 2D
space), data structures such as kd-trees allow efficient retrieval of all points
within a given distance of a specified point, and the time complexity can be as
low as O(mlogm) in the average case. The space requirement of DBSCAN, even
for high-dimensional data, is O(m) because it is necessary to keep only a small
amount of data for each point, i.e., the cluster label and the identification of each
point as a core, border, or noise point.

Strengths and Weaknesses:

Strengths:
 DBSCAN uses a density-based definition of a cluster, it is relatively
resistant to noise and can handle clusters of arbitrary shapes and sizes.
DBSCAN can find many clusters that could not be found using K-means.


Weaknesses:

 DBSCAN has trouble when the clusters have widely varying densities. It also
has trouble with high-dimensional data because density is more difficult to
 define for such data. DBSCAN can be expensive when the computation of
nearest neighbours requires computing all pairwise proximities, as is usually
the case for high-dimensional data.
 DBSCAN

DBSCAN is the abbreviation for Density-Based Spatial Clustering

of Applications with Noise. It is an unsupervised clustering algorithm.DBSCAN
clustering can work with clusters of any size from huge amounts of data and
can work with datasets containing a significant amount of noise. It is basically
based on the criteria of a minimum number of points within a region.

DBSCAN algorithm can cluster densely grouped points efficiently into one
cluster. It can identify local density in the data points among large datasets.
DBSCAN can very effectively handle outliers. An advantage of DBSACN over
the K-means algorithm is that the number of centroids need not be known
beforehand in the case of DBSCAN. DBSCAN algorithm depends upon two

parameters epsilon and minPoints. Epsilon is defined as the radius of each data

point around which the density is

considered.
minPoints is the number of points required within the radius so that the data
point becomes a core point.

In the above figure, we can see that point A has no points inside epsilon(e)
radius. Hence it is a Noise Point. Point B has minPoints(=4) number of points
with epsilon e radius , thus it is a Core Point. While the point has only 1 (
less than minPoints) point, hence it is a Border Point.
Steps Involved in DBSCAN Algorithm.
 First, all the points within epsilon radius are found and the core points
are identified with number of points greater than or equal to minPoints.
 Next, for each core point, if not assigned to a particular cluster, a new
cluster is created for it. All the densely connected points related to the
 core point are found and assigned to the same cluster. Two points are
called densely connected points if they have a neighbor point that has
both the points within epsilon distance. Then all the points in the data
are iterated, and the points that do not belong to any cluster are
marked as noise.


Grid Based Clustering Methods

We can use the grid-based clustering method for multi-resolution of grid-
based data structure. It is used to quantize the area of the object into a
finite number of cells, which is stored in the grid system where all the
operations of Clustering are implemented. We can use this method for its
quick processing time, which is generally independent of the number of
data objects, still dependent on only the multiple cells in each dimension
in the quantized space. 1.STING 2.CLIQUE