Deep Learning
Lecture 05+06: Transfer Learning & Pretrained Models
Presenter: Dr Shahbaz Khan
Overview
Today: Transfer Learning Next Week:
Imagenet Localization
Alexnet Detection
VGG16
Resnet
Transfer learning
Problem with our developed model
Large Data Requirement Long Training Time
Deep learning models need large datasets Training deep networks from scratch
to generalize well. requires extensive computational
Collecting enough labeled data is expensive resources.
and time-consuming. Models with millions of parameters need
Models trained on small datasets often lead days or weeks to converge, even with
to overfitting. powerful hardware.
Problem with our developed model
Difficulty in Optimization Limited Feature Extraction
Custom models often suffer from
Models built from scratch learn features
vanishing/exploding gradients in deep layers.
from randomly initialized weights.
Finding the right combination of
hyperparameters (learning rate, batch size, etc.) Initial layers take time to learn low-level
is complex. features like edges and textures.
Improper initialization can cause poor Higher layers need to learn high-level
performance, leading to unstable gradients abstract features, which requires deep
architectures and more training data.
.
Image net History
ImageNet is one of the most influential datasets in
the history of computer vision and deep learning.
It has driven remarkable advancements in the field,
particularly in the development of state-of-the-art
models
Image net large visual research challenge
The first ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) was held in 2010, and it became an
annual competition.
Early entries used traditional machine learning
techniques such as support vector machines (SVMs) and
handcrafted features (e.g., SIFT, HOG), achieving around
28.2% error rate.
AlexNet: In 2012, a model called AlexNet by
Alex Krizhevsky, Ilya Sutskever, and Geoffrey
Hinton changed the course of the challenge and
computer vision research.
Alex Net
Alex Net
VGG 16
VGG 16
VGG 19
Residual Networks (ResNet) and Skip Connections
ResNet50
https://keras.io/api/applications/
So, Can we use these
models help?
Yes, Of course, and this is
called as transfer learning
Transfer Learning
A technique where a model trained on one task is
reused or fine-tuned for a different, but related, task.
Pretrained Model: Use a model trained on a large dataset
(e.g., ImageNet for images, BERT for NLP).
Knowledge Transfer: The pretrained model learns general
features and representations (like edges, textures in
images, or semantics in text).
New Task: Adapt the model for a new, often smaller,
dataset (e.g., medical image classification)
Feature Extraction
Freeze
Feature Extraction
This is extremely useful when:
•The task of interest has less data.
•But a related task has abundant data.
This is how it works:
•Train a neural network model (base model) on the related task.
•Replace the last few layers on the base model with new layers.
Fine Tuning
Freeze
Fine Tuning
Fine-tuning involves updating the weights of some or all layers of the pre-trained model to
adapt it to the new task.
Unfreeze some layers of the pretrained model and retrain them for the new task..
Solving the Custom Model Problem
Faster Training
Instead of training from scratch, we reuse the weights of pretrained models.
Training time is reduced to hours or minutes instead of days because only a few layers need fine-
tuning.
Improved Accuracy
Transfer learning typically results in higher accuracy since the model starts with useful features
instead of random initialization.
Pretrained models already capture low-level and mid-level features like edges, textures, and
shapes, so fine-tuning focuses on high-level task-specific features.
Thank You
Any Questions?
