UNIT III

NEURAL NETWORKS AND DEEP LEARNING FOR NLP

PART A
COURSE OBJECTIVE: To state the concept of neural network and deep learning
COURSE OUTCOME : To develop the ability to understand neural networks and deep
learning.
Bloom’s Taxonomy Level - K1, K2, K3

S. Question BTL
No
1. Define Feed Forward Neural network. K1
Feedforward Neural Networks (FNNs) are a type of artificial neural
network where the information moves in one direction forward from
the input layer to the output layer, without any loops. These networks
are often used for supervised learning tasks like classification and
regression.
2. Define key concepts of Pattern Storage and Retrieval with examples. K1
 Weights as Memory: The weight values in the neural network
act as the "memory" for the patterns. These weights are
modified during training and are later used to recognize and
recall patterns during retrieval.
Activation Functions: The neurons in a feedforward neural network
apply an activation function (e.g., sigmoid, ReLU, etc.) to compute
their outputs. The activation function determines how the network
responds to the input data and plays a key role in encoding and
retrieving patterns.
 Overfitting: One issue that may arise when training a
feedforward neural network to store patterns is overfitting,
where the network memorizes the training data too well and
performs poorly on unseen data. This can hinder the retrieval
of general patterns that are similar to the stored ones.
3. What are the challenges in Pattern Storage and Retrieval? K2
 Limited Capacity: Feedforward neural networks have a
limited capacity for storing patterns, especially when the
number of patterns becomes large. The network may struggle
to retrieve patterns accurately if it has been trained on too
many inputs or the input space is too large.
 Noise Sensitivity: If an input pattern is noisy or incomplete,
the network may have difficulty retrieving the correct pattern.
 Techniques like regularization and early stopping during
training can help mitigate this issue, but noise remains a
challenge.
 Error Propagation: In deep feedforward neural networks,
errors in early layers can propagate through the network,
affecting the accuracy of pattern retrieval. This is addressed
through the backpropagation algorithm and by using
techniques like dropout or batch normalization.

4. Define Hopfield Model and its key concepts. K1

The Hopfield model is a type of recurrent artificial neural network
(ANN) that was introduced by John Hopfield in 1982. It is primarily
used to model associative memory, where the network can retrieve
patterns based on partial or noisy inputs. It is often referred to as a
"content-addressable" memory system because it can store and
retrieve information by providing part of the input.
Key aspects of the Hopfield model:
Network Structure: The Hopfield network consists of binary neurons,
where each neuron can take one of two states, usually \ (+1 \) or \( -
1 \) (or alternatively, 0 and 1). - The neurons are fully connected,
meaning each neuron is connected to every other neuron in the
network. - The network is symmetric, meaning the connection
weights are the same in both directions (\(w_{ij} = w_{ji} \)). - There
is no self-connection, so the diagonal elements of the weight matrix
are zero (\(w_{ii} = 0 \)).
5. Define Long Short-Term Memory Networks and its key features. K1
Long Short-Term Memory Networks (LSTMs) Long Short-Term
Memory (LSTM) networks are a type of Recurrent Neural Network
(RNN) designed to better capture long-term dependencies in
sequential data. They address the limitations of traditional RNNs,
particularly the vanishing gradient problem, which makes them
inefficient at learning long-range dependencies.
1. Memory Cell: The core of the LSTM unit is a memory cell,
which is capable of maintaining its state over time. This
allows LSTMs to remember information over long sequences
and avoid the vanishing gradient problem typical of basic
RNNs.
2. Gates: LSTMs use three types of gates to regulate the flow of
information into and out of the memory cell.
6. What are the applications and limitations of Hopfield Model? K2
 Applications:
 Associative Memory: The Hopfield network can recall entire
patterns from partial or noisy input.
 Optimization Problems: The energy function of the Hopfield
network can be used to solve combinatorial optimization
problems by encoding the problem constraints as energy
minimization.
 Pattern Recognition: It can be used for recognizing patterns
by retrieving the closest stored pattern given an input.
Limitations:
The Hopfield network is not ideal for large-scale learning or tasks
that require continuous updates or feedback. - It tends to perform
poorly if too many patterns are stored, as the retrieval can become
unreliable. - The network can only store binary
patterns and cannot handle more complex data representations (such
as real-valued data).
7. What are the limitations of LTSM? K2
 Computationally Expensive: LSTMs can be computationally
intensive due to their complex gating mechanisms.
 Difficulty with Very Long Sequences: While LSTMs handle
long-term dependencies better than traditional RNNs, they
may still struggle with extremely long sequences, especially
when compared to other models like Transformer networks.
 Training Complexity: LSTMs can be difficult to train
effectively, requiring careful tuning of hyperparameters, such
as the number of layers, learning rate, and batch size.
8. What are the advantages of LSTM? K2
Advantages of LSTMs:
 Long-Term Dependencies: LSTMs are capable of learning
long-range dependencies due to their memory cells and gates,
which help mitigate the vanishing gradient problem seen in
vanilla RNNs.
 Flexibility: LSTMs can be applied to a wide range of tasks,
including time series forecasting, natural language processing
(NLP), speech recognition, and video analysis.
 Efficient Gradient Flow: The use of gates helps regulate the
gradients, allowing them to propagate through many time
steps without exploding or vanishing.
9. What are the structure of LSTM?
Structure of an LSTM Unit:
At each time step \( t \), the LSTM has the following components:-
\( h_t \): The hidden state (output) at time step \( t \) -\( c_t \):
The cell state at time step \( t \) - \( x_t \): The input at time step \( t \)
- \( f_t \):
Forget gate activation at time step \( t \) - \( i_t \):
Input gate activation at time step \( t \) - \( \hat{C}_t \):
Candidate cell state at time step \( t \) - \( o_t \): Output gate
activation at time step \( t \)
Equations:
1. Forget Gate: \[f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) \]
Where \( \sigma \) is the sigmoid activation function.
2. Input Gate: \[i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) \]
3. Candidate Cell State: \[ \hat{C}_t = \tanh(W_C \cdot [h_{t-
1}, x_t] + b_C) \]
4. Update the Cell State: \[c_t = f_t \cdot c_{t-1} + i_t \cdot \
hat{C}_t \]
5. Output Gate: \[o_t = \sigma(W_o \cdot [h_{t-1}, x_t] +
b_o) \]
6. Hidden State: \[h_t = o_t \cdot \tanh(c_t) \]
10. What are the variants of LSTM? K2
Bidirectional LSTMs (BiLSTMs): These process the input sequence
in both forward and backward directions, allowing them to capture
context from both past and future states.
Gated Recurrent Units (GRUs): A simplified version of LSTMs,
which merge the forget and input gates into a single update gate.
GRUs are computationally more efficient and perform similarly in
many cases. In summary, LSTMs are powerful models for sequential
data, overcoming many of the challenges that traditional RNNs face
in modeling long-term dependencies. Their structure, based on gates
and memory cells, enables them to process time-series data more
effectively than earlier architectures.
11. What are the applications of LSTM? K2
 Natural Language Processing (NLP): Language translation
(e.g., sequence-to-sequence models) - Sentiment analysis -
 Text generation - Speech-to-text
 Time Series Forecasting: - Stock price prediction - Weather
forecasting - Energy demand prediction
 Speech Recognition: - Converting speech signals to text -
Real-time speech translation
 Video Processing: - Action recognition - Frame prediction and
video captioning.
12. What is the Attention Mechanism in NLP? K1
The attention mechanism helps focus AI and NLP models on the
most relevant portion of the input data. The attention model in NLP
searches for the most relevant information in the source sentences.
This is similar to the human cognitive process of concentrating on
one (or few) elements while ignoring the rest of the information
13. What are the benefits of Benefits of using Attention Mechanism in K2
NLP
Here are some of its benefits or advantages:
 Improves model performance by enabling them to focus on
relevant parts of the input sequence.
 Reduces workload by breaking down lengthy sequences into
smaller manageable components.
 Improves decision-making by enhancing the interpretability
of the AI and NLP models.
14. Differentiate the Scaled dot-product attention and multi-head K3
attention.
Scaled dot-product attention
This is a commonly used type of self-attention that assigns attention
weights for each position in the input sequence. It represents the
input sequence in the form of:
 The query matrix indicates the current position of the input
sequence.
 The key matrix displays all the other positions in the input
sequence.
 The value matrix shows the output information for each
position in the input sequence.
Multi-head attention
This is a variant of the scaled dot-product attention mechanism,
where multiple attention heads in parallel learn to attend to a different
 input representation. This technique divides the input sequence into
multiple representations (or heads), which can compute the attention
weight for each head. The benefit is that each head can focus on a
different aspect of the input.
15. Why is the attention mechanism better than the previous encoder- K2
decoder architecture?
 The encoder processes the input sequence and summarizes the
information into a fixed-length context vector.
 The decoder component is initialized with the context vector,
following which it generates the output information.
Thus, the fixed-length context vector design is designed for
shorter input sequences. Frequently, this technique "forgets" the
earlier part of a longer input sequence after processing the entire
sequence. The attention mechanism overcomes this limitation.
16. What are transformers? K2
Transformers are a model architecture in natural language processing
(nlp) that use attention mechanisms to capture dependencies in input
data. They are considered the standard model in modern NLP and are
known for their ability to understand complex patterns and
dependencies in long sequences of words.
17. What are the challenges of transformer models and tackle them? K2
challenges of transformer models and how to tackle them:
 Computational complexity: Transformer models are very
computationally expensive to train and deploy. This is
because they require a large number of parameters and a lot of
data. To tackle this challenge, researchers are developing new
techniques to make transformer models more efficient.
 Data requirements: Transformer models require a large
amount of data to train. This is because they need to learn the
relationships between words in a sentence. To tackle this
challenge, researchers are developing new techniques to pre-
train transformer models on large datasets.
18. Write short notes on the Transformer architecture. K2
Transformer architecture is a neural network architecture that is
based on attention layers. Transformer models are typically made up
of an encoder and a decoder. The encoder takes the input text as input
and produces a sequence of vectors. The decoder takes the output of
the encoder as input and produces a sequence of output tokens.
19. What is attention layer? K1
Attention layers are a type of neural network layer that allows the
model to learn long-range dependencies between words in a sentence.
The attention layer works by computing a score for each pair of
 words in the sentence. The score for a pair of words is a measure of
how related the two words are. The attention layer then uses these
scores to compute a weighted sum of the input vectors. The weighted
sum is the output of the attention layer.
The input to the attention layer is a sequence of vectors. The output
of the attention layer is a weighted sum of the input vectors. The
weights are computed using the scores for each pair of words in the
sentence.
20. What is decoding? K1
DecodingThe decoder in a transformer model takes a sequence of
vectors as input and produces a sequence of words. The decoder also
consists of a stack of self-attention layers. The self-attention layers
work the same way as in the encoder. The decoder also has an RNN,
which takes the output of the self-attention layers as input and
produces a sequence of output tokens. The output tokens are the
words in the output sentence.

PART B

S. Question BTL
No
1. Explain in detail Feed forward Neural Networks: Pattern storage and K2
retrieval.
2. Describe about Hopfield model in detail. K2
3. Explain the Long Short-Term Memory Networks. K2
4. Describe the concept Attention Mechanisms. K2
5. Explain the topic about Transformers. K2