CH-06 Introduction to Neural Networks

Baikuntha Acharya ([email protected])

Department of Applied Sciences and Chemical Engineering
Tribhuvan University, Institute of Engineering, Pulchowk Campus

Baikuntha Acharya
Senior Lecturer / Deputy Head,
Research Coordinator
Department of Electronics & Computer Engineering,
Sagarmatha Engineering College, Lalitpur
www.linkedin.com/in/baikunth2a
www.baikunthaacharya.com.np

© Baikuntha Acharya ([email protected])

 Before Starting!
Brain Vs Computer

Human Brain Computers

200 billion neurons, 32 trillion synapses 1 billion bytes RAM but trillions of bytes on disk

Arithmetic task: 1 brain = 1/10 pocket

Vision task: 1 brain = 1000 super computers
calculator

Brain wins: Memory of real-world facts Computer wins: Memory of arbitrary details

Energy use: 25W Energy watt: 45-120W (CPU)

Processing speed: 100 Hz Processing speed: 109 Hz

Parallel, Distributed Serial, Centralized

Fault Tolerant Generally, not Fault Tolerant

Learns: Yes Learns: Some

Intelligent/Conscious: Usually Intelligent/Conscious: Generally, No

2
© Baikuntha Acharya ([email protected])
 Introduction to Neural Networks

✓ What is a neuron?
• A neuron is a nerve cell that transmits information through electrical and
chemical signals. It is the basic unit of the nervous system.
• Consists of a cell body, dendrites (receive signals), and an axon (sends signals).

• Communicates via synapses using neurotransmitters.

✓ What is artificial neural network?

• “. . . Neural nets are basically mathematical models of information processing . . .”

• “. . . (neural nets) refer to machines that have a structure that, at some level,
reflects what is known of the structure of the brain . . .”

• “A neural network is a massively parallel distributed processor . . . “

✓ Artificial Neural Systems are called:

• neurocomputers

• neural networks

• parallel distributed processors

3
© Baikuntha Acharya ([email protected])
 Introduction to Neural Networks
Brief History of ANN

✓ Creation:
• 1890: William James - defined a neuronal process of learning

• 1911: Ramon y Cajal (1911) introduced the idea of a neuron (brain cell)
Promising Technology:
• 1943: McCulloch and Pitts - earliest mathematical models

• 1949: Hebbian learning by Donald Hebb - 'Neurons that fire together, wire together'

• 1954: Hebb and IBM research group - earliest simulations

• 1958: Frank Rosenblatt - The Perceptron

✓ Disenchantment:
1969: Minsky and Papert - perceptrons have severe limitations (non-linear XOR problem)

✓ Re-emergence:
• 1985: Multi-layer nets that use back-propagation

• 1986: PDP Research Group - multi-disciplined approach

4
© Baikuntha Acharya ([email protected])
 Introduction to Neural Networks
Biological Vs Artificial Neuron

✓ ANNs are inspired by the structure of neurons in our human brain.

• Neurons connect to other neurons by synapses, and the neurons will “fire” which
sends electrochemical activity to neighboring neurons across synapses
• A neuron receives input from other neurons (generally thousands) from its synapses

• Inputs are approximately aggregated by summing

• When the input exceeds a threshold the neuronsends

an electrical spike that travels that travels from the
body, down the axon, to the next neuron(s)


W1
Neuron
I W2
N
P Σ f(n) Outputs
U
T
S Wn-1 Activation

Wn Function

5
© Baikuntha Acharya ([email protected])
 Introduction to Neural Networks (Cont..)

✓ An artificial neural network (ANN) is a computational model

inspired by the human brain, designed to recognize patterns and
make decisions based on data.
• ANNs consist of layers of interconnected nodes (neurons). A typical network
includes an input layer, one or more hidden layers, and an output layer.

✓ Mathematical Formulation: 𝑌 = 𝑓 𝑊𝑋 + 𝑏
• where 𝑋 is the input vector, 𝑊 is the weight matrix, 𝑏 is the bias vector, and 𝑓 is the activation
function applied element-wise.

6
© Baikuntha Acharya ([email protected])
 Introduction to Neural Networks (Cont..)
ANN Matrix Representation

7
© Baikuntha Acharya ([email protected])
 Components of ANN

1. Neurons (Nodes): Basic units of a neural network. Each neuron

receives inputs, processes them, and produces an output.

2. Layers: processes inputs and passes meaningful features to the

next layer. There are three types of layers:
• Input Layer: This layer receives the raw data or features (like pixel values in an
image or words in a sentence). Each neuron in the input layer represents one
feature.

• One or More Hidden Layers: Layers between the input and output layers that
process the data. These layers contain neurons that apply weighted sums and
activation functions to the inputs they receive.

• Output Layer: Produces the final result (e.g., a classification label or a predicted
value). The number of neurons in this layer depends on the number of possible
outputs.

8
© Baikuntha Acharya ([email protected])
 Components of ANN (Cont..)

3. Weights: Each connection between neurons has a weight that determines

the importance of the signal. Weights are adjusted during training to
improve the model’s performance.

4. Bias: A constant value added to the input of each neuron to allow the
network to better fit the data.

5. Activation Function: A function applied to the output of each neuron to

introduce non-linearity, enabling the network to learn complex patterns.

9
© Baikuntha Acharya ([email protected])
 Activation Function
✓ Activation functions are mathematical equations applied to a
neuron's output to introduce non-linearity, enabling neural
networks to learn complex patterns and make decisions.
• Introduce Non-Linearity to ML Models:
• Prevents networks from collapsing into simple linear models.

• Differentiability for Backpropagation Learning:

• Essential for backpropagation in training.

• Output Shaping (Different Range for Different Problem):

• Adjusts the range and distribution of outputs (E.g.: 0-1 for classification, −∞ to +∞ for
regression)

10
© Baikuntha Acharya ([email protected])
 Activation Function (Cont..)
Common Activation Functions

11
© Baikuntha Acharya ([email protected])
 Working of ANN
How does ANN work?

Video source: 3Blue1Brown YouTube Channel 12

© Baikuntha Acharya ([email protected])
 Working of ANN (Cont..)

✓ The general workflow of ANN is:

• Forward Propagation: During this phase, data is passed through the network
from the input layer to the output layer. Each neuron processes the inputs by
multiplying them with the weights, adding the bias, and applying the activation
function to produce an output using the formula:
𝑌 = 𝑓 𝑊𝑋 + 𝑏

• Loss Function: After forward propagation, the network’s output is compared to

the true value using a loss function (e.g., mean squared error for regression or
cross-entropy for classification). This calculates the error, which will guide
adjustments in the next step.

• Backpropagation: This is the learning phase where the error from the output is
propagated backward through the network. The goal is to minimize the error by
adjusting the weights using optimization algorithms like Gradient Descent.
• The gradient (the rate of change of the loss) is calculated with respect to each weight,
and the weights are updated to minimize the loss.
13
© Baikuntha Acharya ([email protected])
 Training of ANN

Concepts:
• Epoch: One complete cycle through the entire training dataset.

• Batch: A subset of data used in one iteration of training.

• Learning Rate: determines how much the weights are adjusted during each step of training

1. Data Preprocessing: Normalize or standardize, train-test split.

2. Model Architecture Design: Choose number of hidden layers, activations.

3. Forward Propagation: 𝑜𝑢𝑡𝑝𝑢𝑡 = 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛_𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛(𝑊𝑒𝑖𝑔ℎ𝑡 . 𝑖𝑛𝑝𝑢𝑡𝑠 + 𝑏𝑖𝑎𝑠)

1
4. Loss Computation: Calculate Mean Squared Error (MSE) = σ 𝑦 − 𝑦ො 2
𝑛
5. Parameter Optimization using Backpropagation:
• Compute gradients of the loss with respect to each parameter.

θ ← θ − η ⋅ ∇θMSE

θ represents the model parameters (weights and biases), η is the learning rate.

• Weight Update: Update parameters using gradient descent (or its variants).

6. Evaluation: Monitor performance on a validation set using metrics like RMSE or 𝑅 2 .

14
© Baikuntha Acharya ([email protected])
 Training of ANN (Cont..)
Backpropagation Algorithm

✓ Backpropagation is a method for training neural networks by computing the

gradient of the loss function with respect to each weight using the chain
rule, and then updating the weights to minimize the loss.
• Compute gradients of the loss with respect to each parameter and update model weights.
θ ← θ − η ⋅ ∇θMSE
θ represents the model parameters (weights and biases), η is the learning rate.

If you are interested more details:

https://www.3blue1brown.com/lesso
ns/backpropagation-calculus

15
© Baikuntha Acharya ([email protected])
 ANN Architectures
Feedforward Neural Networks (FNN/MLP)

✓ A Feedforward Neural Network (or Multi-Layer Perceptron) is a simple

network where data moves in one direction—from input to output through
hidden layers.
• Data flows sequentially through layers without loops.

• Each neuron computes a weighted sum of inputs, adds a bias, and applies a non-linear
activation function.

✓ Key Features:
• Fully connected layers.

• Universal approximator given sufficient complexity.

Output
✓ Applications: Inputs First Hidden Second Hidden Layer
layer Layer

• Classification and regression tasks where input features are independent.

16
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Convolutional Neural Networks (CNN)

✓ CNNs are designed to process grid-like data, such as images, by detecting

local patterns with convolutional filters.
• Convolutional layers slide filters over the input to extract local features.

• Pooling layers reduce dimensionality while retaining key information.

• Fully connected layers combine the learned features for final prediction.

✓ Key Features:
• Hierarchical feature extraction
from simple edges to complex
shapes.

✓ Applications:
• Image and video recognition,
object detection, and
segmentation.

17
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Convolutional Neural Networks (CNN)
✓ Example: Handwritten digit prediction
• CNN architecture for digit recognition. The diagram represents a convolutional neural network (CNN)
processing a 28×28 grayscale image of a handwritten digit.

• Conv_1 & Conv_2: Two convolution layers with a 5×5 kernel and valid padding extract features.

• Max-Pooling: Reduces spatial dimensions using 2×2 pooling after each convolution layer.

• Flattening: Converts feature maps into a single vector for fully connected layers.

• Fully Connected Layers: Two dense layers with ReLU activation and dropout for classification.

• Output Layer: A softmax classifier predicts the digit (0-9)

18
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Recurrent Neural Networks (RNN)
✓ RNNs are designed for sequential data, using loops to retain information
from previous inputs for context.
• Processes sequences by maintaining a hidden state updated at each time step.

• Variants like LSTM and GRU incorporate mechanisms to handle long-range dependencies.

✓ Key Features:
• Captures temporal information, and has memory of past information through internal loops.

✓ Applications:
• Natural language processing (NLP), speech recognition, and time-series forecasting.

19
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Autoencoders

✓ Autoencoders are unsupervised networks that learn to compress input data

into a lower-dimensional space and then reconstruct it.
• An encoder compresses the data into a latent representation.
• A decoder reconstructs the original input from the compressed form.

✓ Applications:
• Data compression, dimensionality reduction and feature extraction, anomaly detection, and
image denoising.

20
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Generative Adversarial Networks (GANs)

✓ GANs consist of two competing networks—a generator and a discriminator

to produce realistic synthetic data.
• The generator creates synthetic samples

• The discriminator evaluates the authenticity of the generated data.

• Both networks are trained adversarially, improving iteratively.

✓ Applications:
• Image synthesis, style transfer, and data augmentation.

21
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Auto-Encoder & GAN – Case Study

https://doi.org/10.3390/pr11020330
22
© Baikuntha Acharya ([email protected])
 ANN Architectures (Cont..)
Transformers

✓ A Transformer is a deep learning model architecture introduced in the

paper "Attention Is All You Need" (2017).

✓ Transformers use self-attention mechanisms to process sequences in

parallel, capturing global context efficiently.
• Self-attention allows every input element to interact with every other.

• Processes entire sequences simultaneously, without relying on recurrence.

✓ Key Features:
• Efficient handling of long-range dependencies.

✓ Transformers models:
• BERT, GPT, and T5

✓ Applications:
• NLP, translation, vision, bioinformatics.
23
© Baikuntha Acharya ([email protected])
 Basic ANN Implementation – Python Keras

✓ What is Keras?
• Keras is an open-source deep learning library that provides a user-friendly API for
building and training neural networks. It runs on top of TensorFlow and simplifies
the process of developing AI models.

• Key Features:
• Easy-to-use and modular.

• Supports multiple backend engines (TensorFlow, Theano, CNTK).

• Provides pre-trained models and tools for deep learning tasks.

✓ Other popular libraries for ML and ANN in Python

24
© Baikuntha Acharya ([email protected])
 Basic ANN Implementation – Python Keras
Keras Implementation Guide

25
© Baikuntha Acharya ([email protected])
 Basic ANN Implementation – Python Keras
Keras Implementation Guide (Cont..)

26
© Baikuntha Acharya ([email protected])
 Basic ANN Implementation – Python Keras
Keras Implementation Guide (Cont..)

27
© Baikuntha Acharya ([email protected])
 Basic ANN Implementation – Python Keras
Minimal Code Example

28
© Baikuntha Acharya ([email protected])
 Advantages of ANN

✓ Through training, the NN learns to solve a problem without the

need for explicit programming
• In fact, while training times might be hours to days, this is far better than the
expert systems that take several man-years.

✓ It can solve many optimization problems.

✓ They are able to handle fuzziness and ambiguity.

✓ They use distributed representations for graceful degradation

• If part of the representation is damaged (destroyed, removed), the performance
degrades “gracefully” rather than completely as with a brittle expert system
which might lack the proper knowledge.

✓ They are capable of supervised & unsupervised learning.

29
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Process Optimization

✓ Process Optimization is the systematic adjustment of operating parameters

(e.g., temperature, pressure, flow rates, reagent concentrations) to improve
key performance metrics such as yield, purity, energy consumption, cost,
etc.
• Enhances efficiency by identifying the best process parameters.

• Reduces energy consumption and raw material waste.

• Improves product quality by minimizing variability.

✓ Example: AI in Reactor Optimization

• AI models analyze historical data from a chemical reactor to optimize temperature and
catalyst concentration.

• Predictive algorithms using ANN adjust setpoints to maximize conversion rates.

• Results in higher yields and lower energy costs.

30
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Process Optimization

✓ Steps in ANN-Based Process Optimization:

1. Define Objectives & Parameters – Identify goals (e.g., maximize yield, minimize
energy) and controllable inputs (temperature, pressure, catalyst, etc.).

2. Data Collection – Gather relevant process data and preprocess.

3. Model Development – Train an ANN model to predict outcomes based on input

conditions.

4. Optimization & Simulation – Evaluate different parameter settings and determine

optimal values.

5. Implementation & Monitoring – Apply optimized conditions and continuously

refine the model based on real-time data

31
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Process Optimization - Example

✓ Optimizing Ethylene Polymerization Reactor

• Objective: Maximize polymer yield while minimizing energy consumption.

• Parameters: Temperature, pressure, and catalyst concentration.

• Data Collection: Gather past reactor data, including yield and operating
conditions.

• Model Development: Train an ANN to predict polymer yield based on input

parameters.

• Optimization: Simulate different settings to find the best combination.

• Implementation: Apply the optimized conditions, monitor yield, and retrain the
ANN for continuous improvement.

32
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Process Optimization – Case Study

✓ AI models analyze molecular structures, train on data, and predict

molecular properties, aiding in the discovery of new chemical
reactions and catalysts.

https://doi.org/10.3390/pr11020330
33
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Predictive Maintenance

✓ AI-driven predictive maintenance analyzes time-series data from sensors

to detect patterns indicating potential failures.
• Reduces unplanned shutdowns and maintenance costs.

• Increases equipment lifespan by addressing issues early.

• Enhances plant safety by preventing catastrophic failures.

✓ Steps in AI-Based Predictive Maintenance

1. Sensor Data Collection – Record vibration, temperature, pressure, and acoustic signals over
time.

2. Time-Series Analysis – Use ANN models ( or variations of ANN - RNNs, LSTMs) to detect
anomalies in sensor readings.

3. Failure Prediction – Develop classification models to determine failure probability.

4. Decision Making – Generate alerts for preventive actions before failure occurs.

5. Implementation – Integrate AI models with maintenance systems for automated scheduling.

34
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Predictive Maintenance - Example

✓ A chemical plant monitors a pump using multiple sensors to predict

potential failure. The time-series sensor data are recorded as:
Vibration Temperature Flow Rate
Timestamp Pressure (bar)
(mm/s) (°C) (m³/h)
2024-01-01 00:00 2.0 80 5.0 100
2024-01-01 06:00 2.1 81 5.0 100
2024-01-01 12:00 2.3 83 5.1 99
2024-01-01 18:00 2.6 85 5.3 98
2024-01-02 00:00 3.0 88 5.5 96

• Trend: Gradual increase in vibration, temperature, and pressure with a slight drop-
in flow rate suggests equipment wear or imbalance.

✓ A predictive model (e.g., LSTM) trained on the data can forecast that
these trends will continue, signaling the need for maintenance.

35
© Baikuntha Acharya ([email protected])
 Applications of ANN in Chemical Engineering
Predictive Maintenance – Case Study

✓ Data driven predictive maintenance system developed for

production lines in manufacturing.

https://doi.org/10.1016/j.eswa.2021.114598 36
© Baikuntha Acharya ([email protected])
 Clustering in ML

✓ Clustering is an unsupervised learning technique that groups data

points into clusters based on similarity, without using labeled
examples. It aims to discover natural structures or patterns in data.
• Similarity Measures: Algorithms typically use distance metrics (e.g., Euclidean,
Manhattan) to determine how close data points are.

• No Predefined Labels: Unlike classification, clustering does not require labeled

data; it finds inherent groupings based solely on the input features.

✓ Popular Methods: K-Means, Hierarchical Clustering, Density-Based

37
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering

✓ K-Means Clustering partitions a dataset into K distinct clusters by

minimizing the variance within each cluster. It does this by iteratively
assigning data points to the nearest cluster centroid and updating the
centroids until convergence.
• Goal: Minimize the within-cluster sum of squares (WCSS).

• Let 𝑋 = {𝑥1, 𝑥2, … , 𝑥𝑛} be the dataset with n data points. The objective of K-Means is to
minimize the following cost function:

𝐾
2
𝐽 = ෍ ෍ 𝑥 − μ𝑖
𝑖=1 𝑥∈𝐶𝑖

Where: 𝐾 = Number of clusters

𝐶𝑖 = Set of points in the i-th cluster

𝜇𝑖 = Centroid of the i-th cluster

∣∣𝑥−𝜇𝑖∣∣2 = Squared Euclidean distance between a point and the centroid

38
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering (Cont..)
Algorithm Steps

1. Initialization:
• Choose k initial centroids randomly (or via methods like k-means for better initialization).

2. Distance Calculation:
• For each data point, calculate distance between datapoints using Euclidean distance.
𝑛
2
𝑑 𝑥, μ𝑖 = ෍ 𝑥𝑗 − μ𝑖𝑗
𝑗=1

3. Cluster Assignment
• Assign each data point to the closest centroid.

4. Update Step:
• Recalculate the centroids as the mean of all points assigned to each cluster.
1
μ𝑖 = ෍ 𝑥
𝐶𝑖
𝑥∈𝐶𝑖

5. Convergence Check:
• Repeat the assignment and update steps until centroids do not change significantly or a
maximum number of iterations is reached.

39
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering (Cont..)
Numerical Example

✓ Use the k-means algorithm and Euclidean distance to cluster the following
8 examples into 3 clusters:
• A1=(2,10), A2=(2,5), A3=(8,4), A4=(5,8), A5=(7,5), A6=(6,4), A7=(1,2), A8=(4,9).

• Suppose that the initial seeds (centers of each cluster) are A1, A4 and A7.

40
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering (Cont..)
Numerical Example (Cont..)

✓ Epoch 1

1. Compute Euclidean Distances

2. New Cluster Assignments:
C1 = C2 = C3 = Closest • C1 → {A1}
Point
(2,10) (5,8) (1,2) Cluster • C2 → {A3, A4, A5, A6, A8}
A1 = (2,10) 0 √13 √65 C1 • C3 → {A2, A7}
A2 = (2,5) √25 √18 √10 C3
3. Compute New Centroids:
A3 = (8,4) √36 √25 √53 C2
New
A4 = (5,8) √13 0 √52 C2 Cluster Points
Centroid
A5 = (7,5) √50 √13 √45 C2
C1 A1 (2,10) (2,10)
A6 = (6,4) √52 √17 √29 C2
A3 (8,4), A4 (5,8), A5
C2 (6,6)
A7 = (1,2) √65 √52 0 C3 (7,5), A6 (6,4), A8 (4,9)
A8 = (4,9) √5 √2 √58 C2 C3 A2 (2,5), A7 (1,2) (1.5, 3.5)

41
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering (Cont..)
Numerical Example (Cont..)

4. Next Epoch - 2:
• Clusters → C1: {A1, A8}, C2: {A3, A4, A5, A6}, C3: {A2, A7}
• Centroids → C1 (3,9.5), C2 (6.5,5.25), C3 (1.5,3.5)

5. Next Epoch - 3:
• Clusters → C1: {A1, A4, A8}, C2: {A3, A5, A6}, C3: {A2, A7}
• Centroids → C1 (3.66,9), C2 (7,4.33), C3 (1.5,3.5)

42
© Baikuntha Acharya ([email protected])
 Unsupervised: K-Means Clustering (Cont..)
How to choose the value of K?

✓ Determining the optimal K is crucial for better grouping:

• Elbow Method
• Plots the sum of squared errors (SSE) vs. the number of clusters.

• The "elbow point" indicates an optimal K, where adding more

clusters yields diminishing returns.

• Silhouette Score
• Measures how well-separated the clusters are.

• A higher silhouette coefficient (ranging from -1 to 1) indicates better clustering.

43
© Baikuntha Acharya ([email protected])
 K-means Clustering Example
Anomaly Detection - Example

✓ Clustering groups similar process data points, allowing detection of

anomalies—data points that deviate significantly from normal.
• Anomalies: Unexpected patterns that indicate potential faults.

✓ K-Means in Anomaly Detection: The key to using K-Means for anomaly

detection is to focus on the concept of distance. Anomalies can be thought
of as data points that are significantly different from the majority of the
dataset.

1. Apply K-Means to the dataset

2. Calculate the distance between each cluster’s point to

their respective cluster’s centroid.

3. Set Threshold – Define a threshold

4. Identify Anomalies – Points exceeding the threshold.

44
© Baikuntha Acharya ([email protected])
 References

✓ Romagnoli, J. A.; Briceño-Mena, L.; & Manee, V. AI in Chemical Engineering: Unlocking

the Power Within Data.

45
© Baikuntha Acharya ([email protected])