Lecture Notes on

ARTIFICIAL NEURAL NETWORKS

Dr.Th.Tangkeshwar Singh
Asstt. Prof.(Senior Scale)
Department of Computer Science M.U.

1
 OUTLINE
INTRODUCTION
HISTORICAL PERSPECTIVE
BIOLOGICAL NEURON & ARTIFICIAL NEURON
SINGLE & MULTI LAYER ARTIFICIAL NEURAL
NETWORKS
TRAINING OF ANNs
PERCEPTRON TRAINING
CLASSIFICATION ABILITY OF PERCEPTRON
PATTERN RECOGNITION MODELS
APPLICATIONS

2
02/02/2023 2
 Introduction
• Artificial Neural Networks(ANN), often just called “
Neural Networks" (NN) are biologically inspired.
• Modeled after the organisation of human brain.
• ANN exhibit such brainlike characteristics as their
ability to learn from experience, generalize on their
knowledge, perform abstraction and make errors,
all more characteristic of human thought than of h
uman made computers.
• An interconnected group of artificial neurons that
uses a mathematical model or computational mo
del for information processing based on a connect
ionist approach.

3
02/02/2023 3
 Historical Perspective

Warren McCulloch and Walter Pitts(1943) )-First to

describe the concept of neural networks.
D.O. Hebb(1949)-Organization of Behavior
Rosenblatt(1959)-Perceptron
Widrow and Holf(1960) –Adaline
Grossberg(1969,1971,1982,1988)-outstar
Fukushima(1975)-Cognitron

4
02/02/2023 4
 Historical Perspective

• John Hopfield(1982)-Hopefield Networks

• Werbos(1974),Parker(1982),Rumelhart,Hinton,Willia
ms(1986)lecum(1986)-Backpropagation Net
• Kohonen’s Self Organising Maps(SOM)(1984)
• Robert Nielse(1987,1988) Grossberg(1988)-
Counterpropagation
• Kosko(1987 )-Bidirectional associative memories
• Carpenter, Grossberg(1987 )-ART
• Fukushima(1984, 1986, 1987)-Neocognitron
• Broomhead &Lowe(1988 )-Radial Basis Fnction
• Vapnik(1990 )-Support Vector Machine

5
02/02/2023 5
6
02/02/2023 6
 Biological neuron & Artificial neuron

Human nervous system

An estimated 10 11 neurons participate in perhaps
10 15 interconnections.

02/02/2023 7
7
Biological Neuron
 Neuron consists of three sections
cell body
dendrites
axon.

8
02/02/2023
Artificial Neuron 8
 Activation Functions
OUT = F(NET)
simple linear function: OUT = 1 if NET > T
OUT = 0 otherwise
where T is a constant threshold value.

02/02/2023
Perceptron Neuron (Frank Resenblatt,1957)9
9
 What are biological NNs?

UNITs: Nerve cells called neurons, many different types

and are extremely complex - around 1011 neurons in the
brain (depending on counting technique) participate in
perhaps 10 15 interconnections.

INTERACTIONs: Signal is conveyed by action potentials,

interactions could be chemical (release or receive
neurotransmitters) or electrical at the synapse

STRUCTUREs: Feedforward and feedback and self-

activation recurrent

10
02/02/2023 10
 The general artificial neuron model has five components :
A set of inputs, xi.
1. A set of weights, wij.

2. A bias, bi.
3. An activation function, f.
4. Neuron output, y
The subscript i indicates the i-th input or weight.
As the inputs and output are external, the
parameters of this model are therefore the
weights, bias and activation function and thus
define the model
02/02/2023 11
11
 Review: Artificial Neural Networks

(ANNs)
A network with interactions, an attempt to mimic the
brain
UNITs: Artificial neuron (linear or nonlinear input-
output unit), small numbers, typically less than a
few hundred
INTERACTIONs: Encoded by weights, how strong a
neuron affects others
STRUCTUREs: Feedforward, Feedback or Recurrent
It is still far too naïve as a brain model and an
information processing device. The actual
“intelligence” exhibited by the most sophisticated
ANNs
02/02/2023
is below the level of a tapeworm. 12
12
 Activation Functions
Squashing function:
Logistic function or Sigmoidal (S-shaped)
F(x)= 1/(1+e –NET)
TANH(N) =>In Matlab,TANSIG(N) calculates its output
according to: n = 2/(1+exp(-2*n))-1

13
02/02/2023 13
14
 Neuron Bias
Desirable to provide each neuron with a trainable bias in
many cases.

Offsets the origin of the logistic function producing an

effect that is similar to adjusting the threshold of the
neuron,thereby permitting more rapid convergence of the
training process.

A weight connected to +1 is added.

The weight is trainable in the same way as all of the other

weights , except that the source is always +1 instead of
being the output of a neuron in a previous layer.

15
02/02/2023 15
16
02/02/2023 16
17
02/02/2023 17
 Single Layer Artificial Neural Network

18
02/02/2023 Multioutput Perceptron 18
 Single Layer Artificial Neural Network
Bias: input = 1
x0 = 1
b1
x1 w11 m
y1
w21 yi  f(  wij x j  bi )
x2 j 1

wm1

yi
xm
19
02/02/2023 19
 Two-Layer Artificial Neural Network

20
02/02/2023 20
 Non-Recurrent Networks OR Feedforward Networks
No feedback connections, have no memory, their
output is solely determined by the current inputs
and the values of the weights.

Recurrent Networks
contain feedback connections
have memory
In some configurations, recurrent networks
recirculate previous outputs back to inputs,
hence output is determined both by their current
input and their previous outputs.
exhibit properties very similar to short term
memory in human.

21
02/02/2023 21
 Training of ANNs
Supervised Training
Requires a teacher
Supervised Training requires the pairing of each
input vector with a target vector representing the
desired output;together these are called training
pair. Network is trained over a number of such
training pairs.
An input is applied, the output of the network is
calculated and compared to the corresponding
target vector, and the difference (error) is fed
back through the network and weights are
changed according to an algorithm that tends to
minimize the error.
The vectors of the training set are applied
sequentially and errors are calculated and
weights adjusted for each vector until the error
for the entire training set is a an acceptably low
level.
22
02/02/2023 22
 Unsupervised Training
(Hebbian Learning - D.O. Hebb,1949,1961)
Requires no teacher
It requires no target vector for the outputs and
hence no comparisons to predetermined ideal
responses.Training set consists solely of input
vectors.
Supervised training has been criticized as being
biologically implausible.
How could the brain of an infant accomplish the
self organization that has been proven to exist in
early development?
Unsupervised training is far more plausible
model of learning in the biological system.
The training process extracts the statistical
properties of the training set and groups similar
vectors into classes.
23
02/02/2023 23
 Hebbian Learning
The Hebb’s postulate
• When an axon of cell A is near enough to excite a cell B and
repeatedly or persistently takes part in firing it, some growth
process or metabolic change takes place in one or both cells
such that A’s efficiency, as one of the cells firing B,
is increased

A
24
02/02/2023 24
 Hebb Rule

A model for unsupervised learning in which

the synaptic strength(weight) was increased if
both the source and destination neuron was
activated.
Hebbian learning will increase its network
weights according to the product of the
excitation levels of the source and destination
neurons. In Symbols
Wij(n+1) = Wij(n) +α OUTi OUT j
α =learning rate coefficient
OUTi = output of neuron i and input to neuron j
25
02/02/2023
OUT j =output of neuron 25
j
 Reinforcement learning(Sutton and Barto,1998)
Concerned with the problem of finding suitable
actions to take in a given situation in order to
maximize a reward.
A teacher is also assumed to be present, but the
right answer is not presented to the network, in
contrast to supervised learning,but the learning
algorithm must instead discover them by a
process of trial and error.
Typically,there is a sequence of states and
actions in which the learning algorithm is
interacting with its environment.
In many cases, the current action not only affects
the immediate reward but also has an impact on
the reward at all subsequent time steps.
trade-off between exploration and exploitation -
active area of machine learning research.
26
02/02/2023 26
 Review:Learning Rules

A learning rule is a procedure for modifying the

weights and biases of a neural network

• It is often called a training algorithm

• The purpose of the learning rule is to train the
network to perform some task

27
02/02/2023 27
 Three categories of learning algorithms
• Supervised learning
- Training set {p1, t1}, {p2, t2}, ….., {pQ, tQ}
- Here tq is called the target for input pq
• Reinforcement learning
- No correct output is provided for each network
input but a measure of the performance is
provided
- It is not very common
• Unsupervised learning
- There are no target outputs available

28
02/02/2023 28
 Perceptron Training(Rosenblatt,1962)
Example:Train the perceptron so that applying a set
of inputs representing an odd number always turns
the light on, while it remains off for even numbers

29
02/02/2023 Perceptron Image-Recognition System 29
 Training Method:
1. Apply an input pattern and calculate the output Y
2.
a. If the output is correct,go to step 1

b. If the output is incorrect, and is zero, add each

input to its corresponding weight; OR

c. If the output is incorrect and is one, subtract

each input from its corresponding weight.
3. Go to step 1.

30
02/02/2023 30
 Delta Rule:
δ =(T-A)
Where T= target output
A=actual output
Step 2b corresponds to δ > 0
Step 2c corresponds to δ < 0
• Δi =η δ xi
where η=learning rate coefficient
• Wi(n+1) = Wi(n) + Δi
Where ΔI = correction associated with ith input Xi
Wi(n+1) = value of weight i after adjustment
Wi(n) = value of weight i before adjustment

31
02/02/2023 31
 Perceptron Learning Rule

Proof of convergence
• We need to show the learning rule will always
converge to weights that accomplish the desired
classification assuming that such weights exist

• We need to show that we only need to update the

weights finite number of times

32
02/02/2023 32
 An Illustrative Example

33
02/02/2023 33
34
02/02/2023 34
 Review: A Neuron

35
02/02/2023 35
 Apple/Banana Example
Training Set
 –
1   1 
   
p
1= t
11 1=  p
2= t
12 0= 
   
 –
1   –
1 

Initial Weights
W
=
0.5
–1
–0.5b=0.5

First Iteration
 –
1 
 
a
=h
ar
d
li
m
Wp+
1h
b=
ar
d
li
m0.5
–
1
– 1
+
0.5
0.5
 
 1
–

a
=
h
a
r
d
l
i
m
–
0.5
=
0

36
02/02/2023 36
 Second Iteration

1
ah
=a
rd
l
imW
( p
2+
b)=h
ar
dl
im(–
0.5
0
– +
1.5
1 
1.5
)
–1

a
=h
a
r
dl
i
m(
2.5
)
=1

37
02/02/2023 37
 Check

–
1
a
=h
ar
d
li
m(W
p+
1b
)h
=a
r
dl
i
m (
–1.5
–
1
– +
0.5
0.5
1 )
–
1

a
==
h
ar
d
l
i
m(
1.5
)1=
t
1

1
a
=h
ar
d
li
m(W
p+
2b
)h
=a
r
dl
i
m (
–1.5
–
1
– +
0.5
0.5
1 )
–
1

a
=
h
a
rd
l
i
m
(–
1.5
)
=
0=
t
2

38
02/02/2023 38
 Linear Separability

Consider perceptron with

39
02/02/2023 W1=1, w2=1 and threshold=1.5 39
 Classification: To classify two linearly separable
classes AND gate function

40
02/02/2023 40
 Perceptron Limitation:
Limitations of perceptron –published by Marvin
Minsky and Seymour Paper(1969)
A single layer perceptron cannot simulate a
simple Exclusive OR function.
The reason for being that Exclusive OR function
is linearly inseparable.
Perceptrons could represent linearly separable
functions but not linearly inseparable functions.

41
02/02/2023 41
 Classification ability of Perceptron
Multilayer Perceptron(MLP):
Different learning algorithms can be used for
MLPs.

42
02/02/2023 42
 Classification ability of Perceptron

43
02/02/2023 43
 Classification ability of Perceptron

44
02/02/2023 44
 PATTERN RECOGNITION MODELS

Approach Representation Recognition function Typical criterion

Template Samples, pixels Correlation Classification

matching curves distance measure error

Statistical Features Discriminant function Classification error

Syntactic Primitives Rules, grammar Acceptance error

Or Structural

Neural Samples, pixels Network function Mean square

Networks features error

45
02/02/2023 45
 EXAMPLE: A NEURAL CHARACTER
RECOGNISER

46
02/02/2023 46
 Applications

Pattern Recognition and Machine Learning

Expert Systems
Speech Recognition
Optical Character Recognition(OCR)
Knowledge Bases, Data Mining
Bomb Detectors
Data vIsualisations
Financial Market Predictions
Medical Diagnosis, i.e. storing medical records
based on case information
Speech production: reading text aloud (NETtalk)

47
02/02/2023 47
 Vision: face recognition , edge detection, visual
search engines
Business,e.g.. rules for mortgage decisions are
extracted from past decisions made by
experienced evaluators, resulting in a network
that has a high level of agreement with human
experts.
Financial Applications: time series analysis,
stock market prediction
Data Compression: speech signal, image, e.g.
faces
Game Playing: backgammon, chess, ...
and Much Much More ...

48
02/02/2023 48
 Books and References

• Neural Computing Theory and Practice

by Philip D Wasserman,Van No Strand
Reinhold,New York,1989
• Pattern Recognition and Machine Learning
by Christopher M. Bishop ,Springer.
• Neural Networks for Pattern Recognition
by Christopher M. Bishop. Oxford University
• Pattern Recognition and Neural Networks
by Brian D. Ripley. Cambridge University
Press. Jan 1996. ISBN 0 521 46086 7.
• Neural Networks, Prentice Hall, 1994
49
02/02/2023 S.Haykin 49
 • Pattern Classification, John Wiley, 2001
R.O. Duda and P.E. Hart and D.G. Stock
• Introduction to the Theory of Neural Computation,
Addison-Wesley Redwood City, 1991
J. Hertz, A. Krogh and R. Palmer
• Neural Networks. An Introduction, Springer-Verlag
Berlin, 1991
B. Mueller and J. Reinhardt
• Introduction to Neural Networks using Matlab
• Related websites
50
02/02/2023 50