UNIT3
NEURAL NETWORKS
MISSION VISION CORE VALUES
CHRIST is a nurturing ground for an individual’s Excellence and Service Faith in God | Moral Uprightness
holistic development to make effective contribution to Love of Fellow Beings
the society in a dynamic environment Social Responsibility | Pursuit of Excellence
CHRIST
Deemed to be University
UNIT2- UN SUPERVISED LEARNING
Neural Network Representation(T2-4.2)
– Problems ( T 2 - 4 . 3 )
– Perceptron's (T2-4.4)
– Multilayer Networks((T2-4.5)
- Back Propagation Algorithms(T2-4.6)
– Advanced Topics(T2-4.8)
Excellence and Service
CHRIST
Deemed to be University
ANN INTRODUCTION
Excellence and Service
CHRIST
Deemed to be University
Biological Neuron Vs Artificial Neuron
Excellence and Service
CHRIST
Deemed to be University
ANN INTRODUCTION
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Artificial Neural Network(ANN) MODEL
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
ANN
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Parameter learning vs Structured learning
● Parameter learning focuses on optimizing the values of parameters within
a predefined model structure to best fit the data, such as adjusting weights
in neural networks or coefficients in regression models.
● parameter learning fine-tunes the model's performance within a given
structure
● Structured learning, on the other hand, involves learning the structure of
the model itself, which may include determining the relationships between
variables, the model architecture, or the network topology.
● Structured learning discovers the optimal model structure to represent the
underlying data patterns accurately
Excellence and Service
CHRIST
Deemed to be University
• Activation functions, helps to learn complex patterns and relationships in
the non-linear data.
• Without activation functions, neural networks would be limited to linear
transformations, severely restricting their capability.
• They help determine the output of neurons, allowing the network to make
more accurate predictions.
• Common activation functions like ReLU, sigmoid, and tanh each have
unique properties that influence the network's performance, convergence
speed, and ability to handle gradient issues.
Excellence and Service
CHRIST
Deemed to be University
Purpose of an activation function is to transform the
summed weighted input from a node into an output value
that is passed on to the next hidden layer or used as the
final output.
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Function may be
summation or
addition
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
(0 to 1)
(-1 to +1)
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
0
CHRIST
Deemed to be University
-1
Excellence and Service
CHRIST
Deemed to be University
Mc-culloch-Pitts Neuron
Excellence and Service
CHRIST
Deemed to be University
Ex1: Sigmoid Activation Function
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Binary Sigmoid value is between 0 to 1
By thresholding we get 0 or 1
Bipolar value ranges from -1 to +1
Excellence and Service
CHRIST
Deemed to be University
Example2: Mc-culloch-Pitts Neuron
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Perceptron Learning
The perceptron model starts by multiplying every input value and its weights. Then,
it adds these values to generate the weighted sum.
This weighted sum is then applied to the activation function “f” to get the
anticipated output. The corresponding activation function is also called the step
function
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Perceptron Rule Flow Chart
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Where
S: Training vector
t=testing vector
If test output & expected
output not equal then
update the new weights
Excellence and Service
CHRIST
Deemed to be University
Ex4: Apply perceptron neuron rule to classify the given dataset into two
classes ie 1 and -1 using the initial weights as zero
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Initialize
Learning Rate α= 1
b=0
t=target
w1=w2=w3=w4=0
New w1=w1+ ˄w1
Excellence and Service
CHRIST
Deemed to be University
Observe Weights
Excellence and Service
CHRIST
Deemed to be University
• No change in weights & bias. We can stop further iterations
• Now Model is fully trained and ready for testing
• FINAL WEIGHTS ARE Observe Weights
• W1=-2, w2=2, w3=0, w4=2, b=0
Excellence and Service
CHRIST
Deemed to be University
Final Neural Network
Final neural network with these
final weights(2-2,2,0,2) can
classify the given data wrt their
target
Excellence and Service
CHRIST
Deemed to be University
Back Propagation
Algorithm
Excellence and Service
CHRIST
Deemed to be University
What is Backpropagation?
● Backpropagation is the process of neural network training. It is the
method of fine-tuning the weights of a neural network based on the
error rate obtained in the previous epoch (i.e., iteration). Proper
tuning of the weights allows you to reduce error rates and make the
model reliable by increasing its generalization.
● Backpropagation is also called as “backward propagation of errors.”
● It is useful mathematical tool for improving the accuracy of predictions
in machine learning
● In case of feedforward neural network where the nodes never form a
cycle. (No feedback).
Excellence and Service
CHRIST
Deemed to be University
Types of Backpropagation Networks
1. Static back-propagation:
It maps static input to static output where the weights are updated after
all data points(images) have been processed. It is useful to solve static
classification issues like optical character recognition.(ex Digits,
Alphabets etc)
2. Recurrent Backpropagation:
● In which feed forward until a fixed value is achieved. After that, the
error is computed and propagated backward. It adjusts weights based
on sequences, considering temporal dependencies.
● The main difference between both of these methods is: that the
mapping is rapid in static back-propagation while it is non-static in
recurrent backpropagation
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Back Propagation Algorithm Steps
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Repeat the steps 1-4 until the stopping condition is met.
Excellence and Service
CHRIST
Deemed to be University
Steps in BPNN algorithm: Construct FFNN
Excellence and Service
CHRIST
Deemed to be University
Step2: Find Output O4, O5, O6, O7 for Node 4-7
Respectively
Excellence and Service
CHRIST
Deemed to be University
Step3: Calculate Error rate
Excellence and Service
CHRIST
Deemed to be University
Exercise: Backpropagation NN
Excellence and Service
CHRIST
Deemed to be University
Epoch1
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Epoch2
Previous error is -0.19 now
reduced to -0.182
Repeat the above steps till error is minimum or zero and whose weights are
used for the classification
Excellence and Service
CHRIST
Exercise: DIY
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Some Basic Concepts
in
Machine Learning
Excellence and Service
CHRIST
Deemed to be University
GRADIENT DESCENT
or
Delta Rule
Perceptron Learning for Non Linearity
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
GRADIENT DESCENT or Delta Rule
Always look
for Global
Minima
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Error Term
Where Td: Target Output
Od: Calculated Output
Weights are
updated
Weights are
Weights are
decreased as
increased as
slope is positive
slope is negative
Excellence and Service
CHRIST
Deemed to be University
How to Decide Learning Rate
In case of higher learning rate In case of smaller learning rate
it never converges to solution it converges to optimum
solution but takes time. Hence
parameter tuning is required
Excellence and Service
CHRIST
Deemed to be University
Global Minima is
Winner
Excellence and Service
CHRIST
Deemed to be University
Model
Overfitting Vs Underfitting
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Try to cover all points Many points missed All points covered
No best fit line No best fit line Best fit line
Training is ok Training is not ok Training is ok
Testing Not ok Testing Not ok Testing ok
Excellence and Service
CHRIST
Deemed to be University
Final Test
Class Test
Excellence and Service
CHRIST
Deemed to be University
Low Bias & Low variance is accepted
Excellence and Service
CHRIST
Deemed to be University
Bias: measure of Gap between Actual & Predicted
Variance: Measure of scatter within each other data
Excellence and Service
CHRIST
Deemed to be University
Concept learning
● Concept learning in machine learning involves identifying and understanding a
target concept or pattern from a set of examples.
● It aims to distinguish between positive examples (those that fit the concept) and
negative examples (those that do not).
● This process involves generating hypotheses or rules that correctly classify new
instances based on the learned concept.
● Techniques include decision trees, rule-based systems, and neural networks.
● Concept learning is fundamental for tasks such as classification and regression,
where the goal is to generalize from observed data to make accurate predictions
on new, unseen data
Excellence and Service
CHRIST
Deemed to be University
Example: Boolean valued Function
Hypothesis can
be a Boolean
values function
or linear
function or non
linear function
Excellence and Service
CHRIST
Deemed to be University
Error Functions
Which Line fits best?
Excellence and Service
CHRIST
Deemed to be University
Objective
Function
Minimize loss
function
Excellence and Service
CHRIST
Deemed to be University
Types of Loss Functions
Excellence and Service
CHRIST
Deemed to be University
The error is less due to
least outliers
Excellence and Service
CHRIST
Deemed to be University
The error is high due to
more outliers
Excellence and Service
CHRIST
Deemed to be University
Low error
High error
Excellence and Service
CHRIST
Deemed to be University
Probability
distributions
Excellence and Service
CHRIST
Deemed to be University
Hinge loss is obtained due to penalized Data points
Excellence and Service
CHRIST
Deemed to be University
How to decide No of Hidden layers
&
Hidden neurons
Excellence and Service
CHRIST
Deemed to be University
ANN Model
Excellence and Service
CHRIST
Deemed to be University
Neuron Network with No hidden layer is just a
Linear Model
Excellence and Service
CHRIST
Deemed to be University
When do we need hidden layers
Excellence and Service
CHRIST
Deemed to be University
How many Hidden layers required?
Excellence and Service
CHRIST
Deemed to be University
No of Neurons in Hidden layers
Excellence and Service
CHRIST
Deemed to be University
Advanced Topics
● RNN
● Swarm Intelligence
Excellence and Service
CHRIST
Deemed to be University
RNN
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
Excellence and Service
CHRIST
Deemed to be University
RNN Sequence for text Mining
Excellence and Service
CHRIST
Deemed to be University
SWARM
INTELLIGENCE
Bio Inspired
105
Excellence and Service
CHRIST
Deemed to be University
Swarm intelligence
● Swarm intelligence is the collective behavior of decentralized, self-organized
systems, typically made up of simple agents interacting locally with one another and
their environment.
● This phenomenon is observed in nature, such as in ant colonies, bird flocking, fish
schooling, and bee hives.
● The agents follow simple rules, and although there is no centralized control, the
group exhibits complex, intelligent behavior.
● Swarm intelligence is applied in artificial intelligence and robotics, optimizing tasks
like routing, scheduling, and problem-solving by mimicking these natural processes.
Key algorithms include Ant Colony Optimization (ACO) and Particle Swarm
Optimization (PSO).
Excellence and Service
CHRIST
Deemed to be University
SWARM INTELLIGENCE
AI not only inspired by human intelligence, AI is also inspired by swarm
intelligence called as Particle Swarm Optimization(PSO)
● Bio Inspired(Ex. Ants, Birds, Bees)
5 Principles
● Awareness: No collisions
● Autonomy: Self Coordinated(No slaves)
● Solidarity: Collective behavior but Independent
● Scalability: Add members dynamically
● Resilience: Self healing when members are removed
APPLICATIONS: Military Robots
107
Excellence and Service
CHRIST
Deemed to be University
Example1: Bio Inspired Computing
Ant Bee Colony(ABC) Optimization
More
accumulation of
pheromones
Based on high volume of pheromones(Chemicals) ants find shortest path
108
Excellence and Service
CHRIST
Deemed to be University
Ant Colony Optimization(ACO)
109
Excellence and Service
CHRIST
Deemed to be University
Example2: Swarm Intelligence(Auklets)
110
Excellence and Service
CHRIST
Deemed to be University
Demo on Swarm Intelligence
Swarm Robots DEMO
Swarm Robots DEMO
Excellence and Service
CHRIST
Deemed to be University
Future Military Technology
Excellence and Service
CHRIST
Deemed to be University
UNIT3 SUMMARY
● Activation Functions
● Neural Network M o d e l
● Problems
● Perceptron’s l e a r n i n g
● MultilayerNetworks
● Back Propagation Algorithms-Examples
● Delta Rule/Grad Descent
● Error Rate
● Overfitting/UnderFitting
● How to select Hidden layers
● Hypothesis Space
● RNN-Introduction
● Swarm Intelligence
Excellence and Service
CHRIST
Deemed to be University
Digit Classification Video-30 Min
https://www.youtube.com/watch?v=zfiSAzpy9NM
Excellence and Service
