AI VIETNAM

All-in-One Course
(TA Session)

Recurrent Neural Networks

Exercise

Dinh-Thang Duong – TA

Year 2023
 AI VIETNAM
All-in-One Course
(TA Session)

Outline
➢ Review
➢ Financial News Sentiment Analysis
➢ Temperature Forecasting
➢ Question

2
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Multilayer Perceptron
Y0 Y1 Y2
Input 𝜃100 𝜃200
𝜃000 Y00 𝜃101 Y01 𝜃201 Y02
𝜃001 𝜃102 𝜃202
Input layer b 𝜃103 𝜃203
𝜃002
𝜃110 𝜃210
𝜃0 10 𝜃111 𝜃211
𝜃011 y10 y y
Hidden layer 𝜃112 11 𝜃212 12
𝜃012 𝜃113 𝜃213
x1
𝜃020 𝜃120 𝜃220
Output layer 𝜃021
Y20
𝜃121
Y21
𝜃221
Y22
𝜃022 𝜃122 𝜃222
𝜃123 𝜃223
x2 𝜃130 𝜃230
𝜃030 𝜃1 𝜃2
𝜃031 31 31
𝜃1
32 𝜃2
32
𝜃032 y30 y31 y32
Multilayer Perceptron 𝜃133 𝜃233

3
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Text Classification Problem

Text Classfication
1
Input 2
Vectorization
9
My name is Thang. I’m a student. MLP
4 Model

4
Can’t connect the relationship between words
8

4
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Basic Approach Problem
Can’t connect the relationship between words

My name is Thang I’m a student

We want a vector that could represent all properties of a word.

Recurrent Neural Network

5
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Recurrent Neural Networks (RNNs)

Output Y ŷ

Hidden
State
h

Input X x

6
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture
Unit

Output Y ŷ

t=0
Hidden
State
h
0 0 0 0 0 0 0 0

Input X x Randomly initialize value

7
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture
ŷ𝑡
ŷ𝑡 = 𝑎2 (ℎ𝑡 𝑊𝑦 + 𝑏𝑦 )

𝑊ℎ 𝑏𝑦 𝑎2 × 𝑊𝑦

+ Inside hidden
state
ℎ𝑡−1 𝑎1 ℎ𝑡

× +

ℎ𝑡 = 𝑎1 (ℎ𝑡−1 𝑊ℎ + 𝑥𝑡 𝑊𝑥 + 𝑏ℎ )
𝑊𝑥 × 𝑏ℎ

𝑥𝑡 8
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture (from t=1 to t=2)

t=1 t=2 9
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture (One-to-One)

Output Y ŷ

Hidden
State
h

Input X x

10
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture (Many-to-Many)

Output Y ŷ ŷ1 ŷ2 ŷ3 ŷT

Hidden
State
h h1 h2 h3 ... hT

Input X x x1 x2 x3 xT

11
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ RNNs Architecture

Output (optional) ŷ1 ŷ2 ŷ3 ŷ4 ŷ5 ŷ6

RNN Layer ℎ1 ℎ2 ℎ3 ℎ4 ℎ5 ℎ6

Input sequence 𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6
(sequence of tokens)
12
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Long Short-Term Memory (LSTM)

𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7 ... 𝑥98 𝑥99 𝑥100

Long input sequence

𝑦99
?
Can’t remember this
information

𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7 ... 𝑥98 𝑥99 𝑥100

13
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ LSTM Architecture

Output Y ŷ ŷ1 ŷ2 ŷ3 ŷT

Hidden
State
h h1 h2 h3 ... hT

Input X x x1 x2 x3 xT

14
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ LSTM Architecture

𝑐𝑡−1 × + 𝑐𝑡

𝑡 ×
Forget Gate Input Gate Output Gate

𝜎 𝜎 × 𝑡 𝜎
𝑊𝑓
𝑊𝑖 + 𝑊𝐶 𝑊𝑜
+ +
+

× 𝑏𝑓 × 𝑏𝑖 × 𝑏𝐶 × 𝑏𝑜

ℎ𝑡−1 ℎ𝑡

𝑥𝑡 15
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ LSTM Architecture

Forget Gate:
• 𝑓𝑡 = 𝜎(𝑊𝑓 ℎ𝑡−1 , 𝑥𝑡 + 𝑏𝑓 )

Input Gate:
• 𝑖𝑡 = 𝜎 𝑊𝑖 ℎ𝑡−1 , 𝑥𝑡 + 𝑏𝑖
• 𝐶𝑡𝑖 = tanh(𝑊𝐶 ℎ𝑡−1 , 𝑥𝑡 + 𝑏𝑐 )

Cell State:
• 𝐶𝑡 = 𝑓𝑡 𝐶𝑡−1 + 𝑖𝑡 𝐶𝑡𝑖

Output Gate:
• 𝑜𝑡 = 𝜎(𝑊𝑜 ℎ𝑡−1 . 𝑥𝑡 + 𝑏𝑜 )

Hidden State:
• ℎ𝑡 = tanh(𝐶𝑡 )𝑜𝑡 16
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖ Bidirectional LSTM

𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7 ... 𝑥98 𝑥99 𝑥100

𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7 ... 𝑥98 𝑥99 𝑥100

𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7 ... 𝑥98 𝑥99 𝑥100

17
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖Bi-LSTM Architecture

ℎ6 ℎ5 ℎ4 ℎ3 ℎ2 ℎ1

Another LSTM layer

18
 AI VIETNAM
All-in-One Course
(TA Session) Review
❖Bi-LSTM Architecture
Output sequence ŷ1 ŷ2 ŷ3 ŷ4 ŷ5 ŷ6

Backward LSTM ℎ′6 ℎ′5 ℎ′4 ℎ′3 ℎ′2 ℎ′1

Forward LSTM ℎ1 ℎ2 ℎ3 ℎ4 ℎ5 ℎ6

Input sequence 𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6
19
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Coding Exercise
Positive

News: Dow closes more than 300 points

lower as December selloff resumes.
😄
Neutral

😐
Problem Statement: Given
Negative
financial news dataset
(download here), build a
sentiment classifier using
RNN/LSTM/BiLSTM layers. 😔 20
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 1: Import libraries

21
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 2: Read dataset

22
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 2: Read dataset

1. Create a classes dictionary to map class name with its ID.

2. Apply ID mapping to the sentiment column.

23
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 3: Text Normalization

Apply text normalization:

1. Lowercasing
2. Unidecoding
3. Stripping
4. Punctuation Removal
5. Stopwords Removal
According to the company 's updated strategy for the years 2009-2012 ,
6. Stemming
Basware targets a long-term net sales growth in the range of 20 % -40 %
with an operating profit margin of 10 % -20 % of net sales .
df['content'] =
df['content'].apply( accord compani updat strategi year 20092012 baswar target longterm net
lambda p: sale growth rang 20 40 oper profit margin 10 20 net sale
text_normalize(p)).a
stype(str) 24
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 4: Build transform function

25
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 5: Train, val, test split

26
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 6: Implement pytorch datasets and dataloader

27
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 7: Implement model

28
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 7: Implement model

Input text = “hello world”

1 5

Padding

1 5 10 10 10

29
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 7: Implement model

1 5

0.04502351 -0.04007018

0.00151128 0.02874336

0.01764284 0.02772436

-0.0089057 0.00842067

30
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 7: Implement model

31
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 7: Implement model

32
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

From line 1 to line 9: Declare fit() function with the

following parameters:
• model: the pytorch model to use.
• train_loader: the dataloader of train set.
• val_loader: the dataloader of val set.
• criterion: the pytorch loss function.
• optimizer: the pytorch learning algorithm.
• device: ‘cuda’ or ‘cpu’.
• epochs: number of epochs.

33
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

Line 10, 11: Declare two empty lists for storing

losses on train set and val set over epochs. These
will be the return values of our function.

34
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

Line 13, 14, 15, 16: Iterate over each epoch, create
an empty list for storing loss on each iteration and
turn the training mode of the model on.

35
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

Line 17, 18: Iterate over each batch of samples in

train dataloader, convert the input tensor into
correspoding device (CPU or GPU).

Line 20, 21: Clear the previous gradient and

execute the prediction on inputs.

36
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

Line 22, 23, 24, 26: Compute the loss, gradient and
update the computed gradient to the optimizer.
Then, append the computed loss to empty list
declared on line 14.

37
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

Line 28, 29: With a list of loss value of each

iteration, we average them to get a single
representative value for the current epoch.

38
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 8: Create fit function

From line 31 to 39: Evaluate the model on val set,

append the val loss to empty list declared in line
11, print the training result on the current epoch. If
the iteration over epoch ends, return the
train_losses and val_losses.

39
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 9: Training

40
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 9: Training

41
 AI VIETNAM
All-in-One Course
(TA Session) Financial News Sentiment Analysis
❖ Step 10: Create evaluation function

42
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Coding Exercise

Predict future temperature in weather forecasting

43
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Coding Exercise
Problem Statement: Given a dataset recorded temperature per hour (download here) Build a model that
given temperature from the previous 6 hours (including the current one), predict temperature of the next
1 hour.

Hour 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00

Condition

Temperature 32 31 31 30 29 26 25

44
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 1: Import libraries

45
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 2: Read dataset

❖ Columns: Date, Summary, Precip Type, Temperature, Apparent Temperature, Humidity, Wind Speed, Wind Bearing,
Visibility, Loud Cover, Pressure, Daily Summary (12 columns).
❖ Number of samples: 96453.

46
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 2: Read dataset

Consider only these two columns

❖ New Columns: Date, Temperature (2 columns).

❖ Number of samples: 96453.

47
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 2: Read dataset

48
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 2: Read dataset

Not in proper format yet X

Model

49
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 3: Prepare (X, y) samples

Problem Statement: Given temperature from the previous 6 hours (including the current one), predict
temperature of the next 1 hour.

50
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 3: Prepare (X, y) samples: Sliding Window

Sliding window technique:

❖ Iterate through each row in datatable.
❖ Select 7 rows.
❖ Extract:
• X: temperature of the first 6 rows.
• y: temperature of the last row.

Window

51
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 3: Prepare (X, y) samples

52
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 4: Create train, val, test set

53
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 5: Create pytorch datasets and dataloader

54
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 6: Implement model

55
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 6: Implement model

𝑋−𝜇 ❖ 𝜇: mean of X
𝑋𝑛𝑜𝑟𝑚 = ❖ 𝜎: standard deviation of X
𝜎

56
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 6: Implement model

57
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 6: Implement model

58
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 7: Training

59
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 7: Training

60
 AI VIETNAM
All-in-One Course
(TA Session) Temperature Forecasting
❖ Step 8: Evaluation

61
 AI VIETNAM
All-in-One Course
(TA Session) Question

?
62
63