Introduction to deep
learning with
PyTorch
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Scientist
What is deep learning?
Deep learning is everywhere:
Language translation
Self-driving cars
Medical diagnostics
Chatbots
Traditional machine learning: relies on
hand-crafted feature engineering
Deep learning: enables feature learning
from raw data
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
What is deep learning?
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
What is deep learning?
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
What is deep learning?
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
What is deep learning?
Inspired by connections in the human brain
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
What is deep learning?
Inspired by connections in the human brain
Neurons ➡ neural networks
Models require large amount of data
At least 100,000s data points
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
PyTorch: a deep learning framework
PyTorch is
one of the most popular deep learning
frameworks
the framework used in many published
deep learning papers
intuitive and user-friendly
has much in common with NumPy
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Importing PyTorch and related packages
PyTorch import in Python
import torch
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Tensors: the building blocks of networks in PyTorch
Load from list Load from NumPy array
import torch np_array = np.array(array)
lst = [[1, 2, 3], [4, 5, 6]] np_tensor = torch.from_numpy(np_array)
tensor = torch.tensor(lst)
Like NumPy arrays, tensors are
Tensor: multidimensional representations of their
Similar to array elements
Building block of neural networks
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Tensor attributes
Tensor shape Tensor device
lst = [[1, 2, 3], [4, 5, 6]]
tensor.device
tensor = torch.tensor(lst)
tensor.shape
device(type='cpu')
torch.Size([2, 3])
Deep learning often requires a GPU, which,
Tensor data type compared to a CPU can offer:
tensor.dtype parallel computing capabilities
faster training times
torch.int64
better performance
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting started with tensor operations
Compatible shapes Incompatible shapes
a = torch.tensor([[1, 1], a = torch.tensor([[1, 1],
[2, 2]]) [2, 2]])
b = torch.tensor([[2, 2], c = torch.tensor([[2, 2, 4],
[3, 3]]) [3, 3, 5]])
Addition / subtraction Addition / subtraction
a + b a + c
tensor([[3, 3], RuntimeError: The size of tensor a
[5, 5]]) (2) must match the size of tensor b (3)
at non-singleton dimension 1
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting started with tensor operations
Element-wise multiplication ... and much more
Transposition
a = torch.tensor([[1, 1],
[2, 2]])
Matrix multiplication
b = torch.tensor([[2, 2], Concatenation
[3, 3]])
Most NumPy array operations can be
a * b
performed on PyTorch tensors
tensor([[2, 2],
[6, 6]])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Creating our first
neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Scientist
Our first neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
import torch.nn as nn
## Create input_tensor with three features
input_tensor = torch.tensor(
[[0.3471, 0.4547, -0.2356]]
)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
import torch.nn as nn
## Create input_tensor with three features
input_tensor = torch.tensor(
[[0.3471, 0.4547, -0.2356]])
A linear layer takes an input, applies a linear
function, and returns output
# Define our first linear layer
linear_layer = nn.Linear(
in_features=3,
out_features=2
)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
import torch.nn as nn
## Create input_tensor with three features
input_tensor = torch.tensor(
[[0.3471, 0.4547, -0.2356]])
# Define our first linear layer
linear_layer = nn.Linear(in_features=3, out_features=2)
# Pass input through linear layer
output = linear_layer(input_tensor)
print(output)
tensor([[-0.2415, -0.1604]],
grad_fn=<AddmmBackward0>)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our first neural network
import torch.nn as nn
## Create input_tensor with three feature
input_tensor = torch.tensor(
[[0.3471, 0.4547, -0.2356]])
# Define our first linear layer
linear_layer = nn.Linear(in_features=3, o
# Pass input through linear layer
output = linear_layer(input_tensor)
print(output)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting to know the linear layer operation
Each linear layer has a .weight and .bias property
linear_layer.weight linear_layer.bias
Parameter containing: Parameter containing:
tensor([[-0.4799, 0.4996, 0.1123], tensor([0.0310, 0.1537], requires_grad=True)
[-0.0365, -0.1855, 0.0432]],
requires_grad=True)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting to know the linear layer operation
output = linear_layer(input_tensor)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting to know the linear layer operation
output = linear_layer(input_tensor)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting to know the linear layer operation
output = linear_layer(input_tensor)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting to know the linear layer operation
output = linear_layer(input_tensor)
For input X , weights W0 and bias b0 , the linear layer performs
y0 = W0 ⋅ X + b0
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Our two-layer network summary
Input dimensions: 1 × 3
Linear layer arguments:
in_features = 3
out_features = 2
Output dimensions: 1 × 2
Networks with only linear layers are called
fully connected
Each neuron in one layer is connected to
each neuron in the next layer
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Stacking layers with nn.Sequential()
A neural network with multiple layers
# Create network with three linear layers
model = nn.Sequential(
nn.Linear(10, 18),
nn.Linear(18, 20),
nn.Linear(20, 5)
)
Input is passed through the linear layers
Input 10 ➡ output 18 ➡ output 20 ➡ Output 5
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Stacking layers with nn.Sequential()
print(input_tensor)
tensor([[-0.0014, 0.4038, 1.0305, 0.7521, 0.7489, -0.3968, 0.0113, -1.3844, 0.8705, -0.9743]])
# Pass input_tensor to model to obtain output
output_tensor = model(input_tensor)
print(output_tensor)
tensor([[-0.0254, -0.0673, 0.0763,
0.0008, 0.2561]], grad_fn=<AddmmBackward0>)
Output is still not yet meaningful
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Discovering
activation functions
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Scientist
Stacked linear operations
We have only seen linear layer networks
Each linear layer multiplies its respective
input with layer weights and adds biases
Even with multiple stacked linear layers,
output still has linear relationship with input
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Why do we need activation functions?
Activation functions add non-linearity to
the network
A model can learn more complex
relationships with non-linearity
"Pre-activation" output passed to the
activation function
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
Mammal or not?
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
Binary classification task:
To predict whether animal is 1 (mammal) or
0 (not mammal)
Input:
Limbs: 4
Eggs: 0
Hair: 1
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
Binary classification task:
To predict whether animal is 1 (mammal) or
0 (not mammal)
Input:
Limbs: 4
Eggs: 0
Hair: 1
output to the linear layers is 6
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
Binary classification task:
To predict whether animal is 1 (mammal) or
0 (not mammal),
pre-activation is 6,
pass it to the sigmoid,
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
Binary classification task:
To predict whether animal is 1 (mammal) or
0 (not mammal),
we take the pre-activation (6),
pass it to the sigmoid,
and obtain a value between 0 and 1.
Using the common threshold of 0.5:
If output is > 0.5, class label = 1 (mammal)
If output is <= 0.5, class label = 0 (not
mammal)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Meet the sigmoid function
import torch
import torch.nn as nn
input_tensor = torch.tensor([[6.0]])
sigmoid = nn.Sigmoid()
output = sigmoid(input_tensor)
tensor([[0.9975]])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Activation function as the last layer
model = nn.Sequential(
nn.Linear(6, 4), # First linear layer
nn.Linear(4, 1), # Second linear layer
nn.Sigmoid() # Sigmoid activation function
)
Note. Sigmoid as last step in network of linear layers is equivalent to traditional logistic
regression.
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
SIGMOID for BINARY classification problems
SOFTMAX for MULTI-CLASS classification
N=3 classes:
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
SIGMOID for BINARY classification problems
SOFTMAX for MULTI-CLASS classification
N=3 classes:
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
SIGMOID for BINARY classification problems
SOFTMAX for MULTI-CLASS classification
N=3 classes:
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
SIGMOID for BINARY classification problems
SOFTMAX for MULTI-CLASS classification
N=3 classes:
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
takes N-element vector as input and
outputs vector of same size
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
takes N-element vector as input and
outputs vector of same size
outputs a probability distribution:
each element is a probability (it's
bounded between 0 and 1)
the sum of the output vector is equal to 1
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
takes N-element vector as input and
outputs vector of same size
outputs a probability distribution:
each element is a probability (it's
bounded between 0 and 1)
the sum of the output vector is equal to 1
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Getting acquainted with softmax
import torch dim = -1 indicates softmax is applied to
import torch.nn as nn the input tensor's last dimension
nn.Softmax() can be used as last step in
# Create an input tensor
input_tensor = torch.tensor( nn.Sequential()
[[4.3, 6.1, 2.3]])
# Apply softmax along the last dimension
probabilities = nn.Softmax(dim=-1)
output_tensor = probabilities(input_tensor)
print(output_tensor)
tensor([[0.1392, 0.8420, 0.0188]])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
