Introduction to PyTorch
Outline
● Deep Learning
○ RNN
○ CNN
○ Attention
○ Transformer
● Pytorch
○ Introduction
○ Basics
○ Examples
Introduction to PyTorch
What is PyTorch?
● Open source machine learning library
● Developed by Facebook's AI Research lab
● It leverages the power of GPUs
● Automatic computation of gradients
● Makes it easier to test and develop new ideas.
Other libraries?
Why PyTorch?
● It is pythonic- concise, close to Python conventions
● Strong GPU support
● Autograd- automatic differentiation
● Many algorithms and components are already
implemented
● Similar to NumPy
Why PyTorch?
Getting Started with PyTorch
Installation
Via Anaconda/Miniconda:
condainstall pytorch-c pytorch
Via pip:
pip3 install torch
PyTorch Basics
iPython Notebook Tutorial
bit.ly/pytorchbasics
Tensors
Tensors are similar to NumPy’s ndarrays, with the addition being that
Tensors can also be used on a GPU to accelerate computing.
Common operations for creation and manipulation of these Tensors are
similar to those for ndarrays in NumPy. (rand, ones, zeros, indexing,
slicing, reshape, transpose, cross product, matrix product, element wise
multiplication)
Tensors
Attributes of a tensor 't':
● t= torch.randn(1)
requires_grad- making a trainable parameter
● By default False
● Turn on:
○ t.requires_grad_()or
○ t = torch.randn(1, requires_grad=True)
● Accessing tensor value:
○ t.data
● Accessingtensor gradient
○ t.grad
grad_fn- history of operations for autograd
● t.grad_fn
Loading Data, Devices and CUDA
Numpy arrays to PyTorch tensors Fallback to cpu if gpu is unavailable:
● torch.from_numpy(x_train) ● torch.cuda.is_available()
● Returns a cpu tensor!
Check cpu/gpu tensor OR numpyarray ?
PyTorchtensor to numpy
● type(t)or t.type()returns
● t.numpy() ○ numpy.ndarray
○ torch.Tensor
Using GPU acceleration ■ CPU - torch.cpu.FloatTensor
■ GPU - torch.cuda.FloatTensor
● t.to()
● Sends to whatever device (cudaor cpu)
Autograd
● Automatic Differentiation Package
● Don’t need to worry about partial differentiation,
chain rule etc.
○ backward() does that
● Gradients are accumulated for each step by default:
○ Need to zero out gradients after each update
○ tensor.grad_zero()
Optimizer and Loss
Optimizer
● Adam, SGD etc.
● An optimizer takes the parameters
we want to update, the learning rate
we want to use along with other
hyper-parameters and performs the
updates
Loss
● Various predefined loss functions to
choose from
● L1, MSE, Cross Entropy
Model
In PyTorch, a model is represented by a regular Python class that inherits from the Module class.
● Two components
○ __init__(self): it defines the parts that make up the model- in our case, two
parameters, a and b
○ forward(self, x) : it performs the actual computation, that is, it outputs a prediction,
given the inputx
PyTorch Example
(neural bag-of-words (ngrams) text classification)
bit.ly/pytorchexample
Overview
Sentence
Embedding
Layer
Linear
Layer
Softmax
Training Evaluation
Cross
Prediction
Entropy
Design Model
● Initilaize modules.
● Use linear layer here.
● Can change it to RNN,
CNN, Transformer etc.
● Randomly initilaize
parameters
● Foward pass
Preprocess
● Build and preprocess dataset
● Build vocabulary
Preprocess
● One example of dataset:
● Create batch ( Used in SGD )
● Choose pad or not ( Using [PAD] )
Training each epoch
Iterable batches
Before each optimization, make
previous gradients zeros
Forward pass to compute loss
Backforward propagation to
compute gradients and update
parameters
After each epoch, do learning
rate decay ( optional )
Test process
Do not need back propagation or parameter update !
The whole training process
● Use CrossEntropyLoss()
as the criterion. The
input is the output of the
model. First do
logsoftmax, then
compute cross-entropy
loss.
● Use SGD as optimizer.
● Use exponential decay
to decrease learning rate
Print information to monitor
the training process
Evaluation with testdataset or random news
