06-02-2019

CHAPTER 5
UNSUPERVISED
LEARNING NETWORKS

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 1
by: (Chiranji Lal)

UNSUPERVISED LEARNING

• These competitive nets arc those where the weights remain fixed,
even during training process. The idea of competition is used
among neurons for enhancement of contrast in their activation
functions.
• Three such nets are- Maxnet, Mexican hat and Hamming net.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 2
by: (Chiranji Lal)

1
 06-02-2019

• Unsupervised learning is not having supervision of

a teacher.
• The learning process is independent.
• During the training of ANN under unsupervised
learning, the input vector of similar type are
combined to form clusters.
• When the new pattern is applied, then the NN gives
an output response indicating the class to which
input pattern belongs.
• No feedback from the environment.
• In this type of learning the network itself must
discover the patterns, feature from the input data
and the relation for the input data over the output.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 3
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 4
by: (Chiranji Lal)

2
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 5
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 6
by: (Chiranji Lal)

3
 06-02-2019

COMPETITIVE LEARNING
 Output units compete, so that eventually only one neuron (the one
with the most input) is active in response to each output pattern.

 The total weight from the input layer to each output neuron is
limited. If some connections are strengthened, others must be
weakened.

 A consequence is that the winner is the output neuron

whose weights best match the activation pattern.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 7
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 8
by: (Chiranji Lal)

4
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 9
by: (Chiranji Lal)

• The learning algorithm used ·in most of these nets is known as

Kohonen learning.
• In this learning, the units update their weights by forming a new
weight vector, which is a linear combination of the old weight vector
and the new input vector.
• Also, the learning continues for the unit whose weight vector is
closest to the input vector. The weight updation formula used in
Kohonen learning for output cluster unit j is given as:

• where x is the input vector; wj the weight vector for unit j; alpha the
learning rare whose value decreases monotonically as training
continues.
• There exist two methods to determine the winner of the network
during competition: (a) Euclidean distance and (b) dot product.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 10
by: (Chiranji Lal)

5
 06-02-2019

The networks based on

unsupervised learning.
Fixed Weight Networks

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 11
by: (Chiranji Lal)

Maxnet
• This is also a fixed weight network, which serves as a subnet for
selecting the node having the highest input. All the nodes are fully
interconnected and there exists symmetrical weights in all these
weighted interconnections.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 12
by: (Chiranji Lal)

6
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 13
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 14
by: (Chiranji Lal)

7
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 15
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 16
by: (Chiranji Lal)

8
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 17
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 18
by: (Chiranji Lal)

9
 06-02-2019

•Hamming Network
• In most of the neural networks using unsupervised learning, it is
essential to compute the distance and perform comparisons.
This kind of network is Hamming network, where for every
given input vectors, it would be clustered into different groups.
• Following are some important features of Hamming Networks −
• Lippmann started working on Hamming networks in 1987.
• It is a single layer network.
• The inputs can be either binary {0, 1} or bipolar {-1, 1}.
• The weights of the net are calculated by the exemplar vectors.
• (the weight vector for an output unit in a clustering net is exemplar
vector or code book vector for the pattern of inputs, which the net
has placed on that output unit)
• It is a fixed weight network which means the weights would
remain the same even during training.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 19
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 20
by: (Chiranji Lal)

10
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 21
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 22
by: (Chiranji Lal)

11
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 23
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 24
by: (Chiranji Lal)

12
 06-02-2019

UNSUPERVISED LEARNING
 No help from the outside.

 No training data, no information available on the desired output.

 Learning by doing.

 Used to pick out structure in the input:

• Clustering,
• Reduction of dimensionality  compression.

 Example: Kohonen’s Learning Law.

 nodes compete for the right to respond to a
subset of the input data

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 25
by: (Chiranji Lal)

FEW UNSUPERVISED LEARNING NETWORKS

There exists several networks under this category, such as

 Kohonen Self-organizing Feature Maps,

 Learning Vector Quantization,

 Adaptive Resonance Theory.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 26
by: (Chiranji Lal)

13
 06-02-2019

Introduction
 The network by itself should discover any relationship of interest,
such as features, patterns, contours, correlations or categories,
classification in the input data, and thereby translate the
discovered relationship into output.
 Such network are also called as Self-organizing network.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 27
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 28
by: (Chiranji Lal)

14
 06-02-2019

SELF-ORGANIZATION
 Network Organization is fundamental to the brain

• Functional structure.
• Layered structure.
• Both parallel processing and serial processing require organization
of the brain.

• What is Topology Preservation

• Preservation of the neighborhood relation of the observations in the output
space. It means that the observations which are neighbors in the input space
should be projected in neighbor nodes.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 29
by: (Chiranji Lal)

SELF-ORGANIZING NETWORKS
 Discover significant patterns or features in the input data.

 Discovery is done without a teacher.

 Synaptic weights are changed according to local rules.

 The changes affect a neuron’s immediate environment until a final

configuration develops.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 30
by: (Chiranji Lal)

15
 06-02-2019

KOHONEN SELF-ORGANIZING FEATURE MAP

(KSOFM)

 The Kohonen model provides a topological mapping.

 (Preservation of the neighborhood relation of the
observations in the output space. It means that the
observations which are neighbors in the input space should be
projected in neighbor nodes.)
 It places a fixed number of input patterns from the input
layer into output or Kohonen layer.

 Training in the Kohonen network begins with the winner’s

neighborhood of a fairly large size. Then, as training
proceeds, the neighborhood size gradually decreases.

 Kohonen SOMs result from the synergy of three basic processes

• Competition,
• Cooperation,
Original Slides: (Sivanandam & Deepa). Edited
• Adaptation.
2/6/2019
by: (Chiranji Lal)
31

Architecture

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 32
by: (Chiranji Lal)

16
 06-02-2019

ARCHITECTURE OF KSOFM

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 33
by: (Chiranji Lal)

COMPETITION OF KSOFM
 Each neuron in an SOM is
assigned a weight vector with the
same dimensionality N as the
input space.

 Any given input pattern is

compared to the weight
vector of each neuron and the
closest neuron is declared the
winner.

 The Euclidean norm is commonly

used to measure distance.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 34
by: (Chiranji Lal)

17
 06-02-2019

CO-OPERATION OF KSOFM
 The activation of the winning neuron is spread to neurons in its
immediate neighborhood.
• This allows topologically close neurons to become
sensitive to similar patterns.

 The winner’s neighborhood is determined on the lattice topology.

• Distance in the lattice is a function of the number of lateral
connections to the winner.

 The size of the neighborhood is initially large, but shrinks over

time.
• An initially large neighborhood promotes a topology-preserving
mapping.
• Smaller neighborhoods allow neurons to specialize in the latter
stages of training.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 35
by: (Chiranji Lal)

ADAPTATION OF KSOFM
During training, the winner neuron
and its topological neighbors are
adapted to make their weight
vectors more similar to the input
pattern that caused the activation.

Neurons that are closer to the winner

will adapt more heavily than neurons
that are further away.

The magnitude of the adaptation is

controlled with a learning rate,
which decays over time to ensure
convergence of the SOM.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 36
by: (Chiranji Lal)

18
 06-02-2019

KSOFM ALGORITHM

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 37
by: (Chiranji Lal)

EXAMPLE OF KSOFM

Find the winning neuron using the Euclidean distance:

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 38
by: (Chiranji Lal)

19
 06-02-2019

Neuron 3 is the winner and its weight vector W3 is updated

according to the competitive learning rule:

The updated weight vector W3 at iteration (p+1) is

determined as:

The weight vector W3 of the winning neuron 3 becomes closer

to the input vector X with each iteration.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 39
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 40
by: (Chiranji Lal)

20
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 41
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 42
by: (Chiranji Lal)

21
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 43
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 44
by: (Chiranji Lal)

22
 06-02-2019

LEARNING VECTOR
QUANTIZATION
(LVQ)
Supervised or Unsupervised?

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 45
by: (Chiranji Lal)

Supervised or Unsupervised?
• The Learning Vector Quantization algorithm is a
supervised neural network that uses a competitive
(winner-take-all) learning strategy.
• It is related to other supervised neural networks
such as the Perceptron and the Back-propagation
algorithm.
• It is related to other competitive learning neural
networks such as the Self-Organizing Map algorithm
that is a similar algorithm for unsupervised learning
with the addition of connections between the
neurons.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 46
by: (Chiranji Lal)

23
 06-02-2019

LEARNING VECTOR QUANTIZATION (LVQ)

 LVQ is a process of classifying the patterns, wherein each output
unit represents a particular class.

 The output unit weight vector is called the reference vector or

code book vector for the class which the unit represents.

 The LVQ is a classifier paradigm that adjusts the boundaries

between categories to minimize existing misclassifications.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 47
by: (Chiranji Lal)

LEARNING VECTOR QUANTIZATION (LVQ)

 This is a (special case of competitive net which uses)
supervised version of vector quantization. Classes are predefined
and we have a set of labeled data.

 The goal is to determine a set of prototypes that best represent

each class.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 48
by: (Chiranji Lal)

24
 06-02-2019

BASIC SCHEME OF LVQ

Step 1: Initialize prototype vectors for different classes.
Step 2: Present a single input.
Step 3: Identify the closest prototype, i.e., the so-called winner.
(which has weight vector very close to the input vector)
Step 4: Move the winner
• closer toward the data (same class),
• away from the data (different class).

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 49
by: (Chiranji Lal)

LVQ: Flowchart

• In case of
training, a set of
training input
vectors with a
known
classification is
provided with
some initial
distribution of
reference vector.
• Assign initial
weights and
classifications
randomely.
• K-mean
clustering
method.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 50
by: (Chiranji Lal)

25
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 51
by: (Chiranji Lal)

You also read

• VARIANTS OF LVQ
• LVQ 1
• LVQ 2
• LVQ 2.1
• LVQ 3

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 52
by: (Chiranji Lal)

26
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 53
by: (Chiranji Lal)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 54
by: (Chiranji Lal)

27
 06-02-2019

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 55
by: (Chiranji Lal)

ADAPTIVE
RESONANCE THEORY
(ART) NETWORK
• Adaptive Resonance Theory (ART) networks is
always open to new learning (adaptive) without
losing the old patterns (resonance).
• Basically, ART network is a vector classifier which
accepts an input vector and classifies it into one of
the categories depending upon which of the stored
pattern it resembles the most.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 56
by: (Chiranji Lal)

28
 06-02-2019

ART Operating Principal

• Recognition phase − The input vector is compared with the
classification presented at every node in the output layer.
The output of the neuron becomes “1” if it best matches
with the classification applied, otherwise it becomes “0”.
• Comparison phase − In this phase, a comparison of the
input vector to the comparison layer vector is done. The
condition for reset is that the degree of similarity would be
less than vigilance parameter.
• Search phase − In this phase, the network will search for
reset as well as the match done in the above phases. Hence,
if there would be no reset and the match is quite good, then
the classification is over. Otherwise, the process would be
repeated and the other stored pattern must be sent to find
the correct match.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 57
by: (Chiranji Lal)

ADAPTIVE RESONANCE THEORY (ART)

NETWORK

 Adaptive Resonance Theory (ART) is a family of algorithms for

unsupervised learning developed by Carpenter and Grossberg.

 ART is similar to many iterative clustering algorithms where each

pattern is processed by

• finding the "nearest" cluster (a.k.a. prototype or template) to

that exemplar (desired).
• updating that cluster to be "closer" to the exemplar.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 58
by: (Chiranji Lal)

29
 06-02-2019

ARCHITECTURES OF ART NETWORK

 ART1, designed for binary features.

 ART2, designed for continuous (analog) features.

 ARTMAP, a supervised version of ART.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 59
by: (Chiranji Lal)

Fundamental Architecture
• Three groups of neurons are used to build an ART
network.
• Input processing unit (F1 layer)
• Input portion
• Interface portion
• Clustering units (F2)
• Control mechanism (control degree of similarity of patterns place on
the same cluster)

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 60
by: (Chiranji Lal)

30
 06-02-2019

FUNDAMENTAL ALGORITHM OF ART NETWORK

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 61
by: (Chiranji Lal)

ART1
• In ART1 network, it is not necessary to present an input
pattern in a particular order; it can be presented in any
order.
• ART 1 network can be practically implemented by analog
circuits and governing the different equations, i.e., the
bottom-up and top-down weights are controlled by
differential equations.
• ART 1 network runs throughout autonomously.
• It doesn't require any external control signals and can run
stably with infinite patterns of input data.
• ART1 net is trained using fast learning method, in which the
weights reach equilibrium during each learning trial.
• During the resonance phase, the activations of F1 units do
not change; hence the equilibrium weights can be
determined exactly.
Original Slides: (Sivanandam & Deepa). Edited
2/6/2019 62
by: (Chiranji Lal)

31
 06-02-2019

BASIC ARCHITECTURE OF ART1

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 63
by: (Chiranji Lal)

ART1 UNITS
ART1 Network is made up of two units

 Computational units
• Input unit (F1 unit – input and interface).
• Cluster unit (F2 unit – output).
• Reset control unit (controls degree of similarity).

 Supplemental units
• One reset control unit.
• Two gain control units.

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 64
by: (Chiranji Lal)

32
 06-02-2019

ART2 NETWORK

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 65
by: (Chiranji Lal)

BASIC ARCHITECTURE OF ART2

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 66
by: (Chiranji Lal)

33
 06-02-2019

ART2 ALGORITHM

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 67
by: (Chiranji Lal)

SUMMARY
This chapter discussed on the various unsupervised learning networks
like:

 Kohonen Self-organizing Feature Maps

 Learning Vector Quantization

 Adaptive Resonance Theory

Original Slides: (Sivanandam & Deepa). Edited

2/6/2019 68
by: (Chiranji Lal)

34