ECE 498: ST:
Introduction to Applied Machine Learning
• Tao Han, Ph.D.
• Associate Professor
• Electrical and Computer Engineering
• Newark College of Engineering
• New Jersey Institute of Technology
• https://tao-han-njit.netlify.app
Slides are designed based on Prof. Hung-yi Lee’s Machine Learning courses at National Taiwan University
http://weebly110810.weebly.com/3
96403913129399.html
http://www.sucaitianxia.com/png/c
Transfer Learning artoon/200811/4261.html
Dog/Cat
Classifier
cat dog
Data not directly related to the task considered
elephant tiger dog cat
Similar domain, different tasks Different domains, same task
Transfer Learning - Overview
Source Data (not directly related to the task)
labelled
Model Fine-tuning
labelled
Target Data
unlabeled
Warning: different terminology in different literature
Model Fine-tuning
One-shot learning: only a few
examples in target domain
• Task description
• Source data: 𝑥 𝑠 , 𝑦 𝑠 A large amount
• Target data: 𝑥 𝑡 , 𝑦 𝑡 Very little
• Example: (supervised) speaker adaption
• Source data: audio data and transcriptions from many
speakers
• Target data: audio data and its transcriptions of specific
user
• Idea: training a model by source data, then fine-
tune the model by target data
• Challenge: only limited target data, so be careful about
overfitting
Conservative Training
Output layer output close Output layer
parameter close
initialization
Input layer Input layer
Target data (e.g.
Source data
A little data from
(e.g. Audio data of
target speaker)
Many speakers)
Layer Transfer
Output layer Copy some parameters
Target data
Input layer 1. Only train the rest layers (prevent
Source
data overfitting)
2. fine-tune the whole network (if
there is sufficient data)
Layer Transfer
• Which layer can be transferred (copied)?
• Image: usually copy the first few layers
Pixels Layer 1 Layer 2 Layer L
x1 …… ……
x2 …… elephant
……
……
……
……
xN …… ……
Transfer Learning - Overview
Source Data (not directly related to the task)
labelled
Model Fine-tuning
labelled
Multitask Learning
Target Data
unlabeled
Warning: different terminology in different literature
Multitask Learning
• The multi-layer structure makes NN suitable for
multitask learning
Task A Task B
Task A Task B
Input
Input feature Input feature
feature
for task A for task B
Multitask Learning
- Multilingual Speech Recognition
states of states of states of states of states of
French German Spanish Italian Mandarin
Human languages
share some common
characteristics.
acoustic features
Similar idea in translation: Daxiang Dong, Hua Wu, Wei He, Dianhai Yu and
Haifeng Wang, "Multi-task learning for multiple language translation.“, ACL 2015
Multitask Learning - Multilingual
50
Character Error Rate
45
40 Mandarin
only
35
With
30 European
Language
25
1 10 100 1000
Hours of training data for Mandarin
Huang, Jui-Ting, et al. "Cross-language knowledge transfer using multilingual
deep neural network with shared hidden layers." ICASSP, 2013
Transfer Learning - Overview
Source Data (not directly related to the task)
labelled
Model Fine-tuning
labelled
Multitask Learning
Target Data
Domain-adaptation
unlabeled
Warning: different terminology in different literature
You have learned a lot about ML. Training a classifier is
not a big deal for you. ☺
Training
Data
Testing
Data
99.5% 57.5%
The results are from: http://proceedings.mlr.press/v37/ganin15.pdf
Domain shift: Training and testing data have different
distributions. Domain adaptation
Domain Shift
Training Data Testing Data
Source Target
Domain Domain
1 2 3 4 5 1 2 3 4 5
This is “0”. This is “1”.
Source Domain
Domain Adaptation (with labeled data)
“4” “0” “1”
Knowledge of target domain
• Idea: training a model by source data,
“8”
then fine-tune the model by target data
• Challenge: only limited target data, so be Little but
careful about overfitting labeled
Source Domain
Domain Adaptation (with labeled data)
“4” “0” “1”
Knowledge of target domain
“8”
Large amount of Little but
unlabeled data labeled
Basic Idea Learn to ignore colors
Feature
Extractor feature
(network)
Source
The same
Different
distribution
Target
Feature
Extractor feature
(network)
Domain Adversarial Training
image class distribution
Feature Label
“4”
Extractor Predictor
Source
(labeled)
blue points
Target
(unlabeled)
red points
Domain Adversarial Training
𝜃𝑓∗ = min 𝐿 − 𝐿𝑑 always zero?
𝜃𝑓
𝜃𝑓 𝜃𝑝
Feature Label
“4”
Extractor Predictor
𝐿
Generator 𝜃𝑝∗ = min 𝐿
𝜃𝑝
• Feature extractor: Learn 𝜃𝑑∗ = min 𝐿𝑑
𝜃𝑑
𝐿𝑑
to “fool” domain classifier 𝜃𝑑
Domain Source?
• Also need to support Classifier Target?
label predictor
Discriminator
Domain Adversarial Training
Yaroslav Ganin, Victor Lempitsky, Unsupervised Domain Adaptation by Backpropagation,
ICML, 2015
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand,
Domain-Adversarial Training of Neural Networks, JMLR, 2016
class 1 (source) Target data
class 2 (source) (class unknown)
Limitation
Decision boundaries learned
from source domain
Source and target data Target data (unlabeled
are aligned, but …… far from boundary)
Considering Decision Boundary
unlabeled Small entropy
Feature Label
Extractor Predictor
1 2 3 4 5
unlabeled
Large entropy
Feature Label
Extractor Predictor
1 2 3 4 5
Used in Decision-boundary Iterative Refinement Training with
a Teacher (DIRT-T) https://arxiv.org/abs/1802.08735
Maximum Classifier Discrepancy https://arxiv.org/abs/1712.02560
Transfer Learning - Overview
Source Data (not directly related to the task)
labelled
Model Fine-tuning
labelled
Multitask Learning
Target Data
Domain-adaptation
unlabeled
Zero-shot learning
Warning: different terminology in different literature
http://evchk.wikia.com/wiki/%E8%8
Zero-shot Learning D%89%E6%B3%A5%E9%A6%AC
• Source data: 𝑥 𝑠 , 𝑦 𝑠 Training data Different
• Target data: 𝑥 𝑡 Testing data tasks
𝑥 𝑠: …… 𝑥𝑡 :
𝑦𝑠: cat dog …… Alpaca
How we solve this problem?
Zero-shot Learning
• Representing each class by its attributes
Training
1 0 0 1 1 1 Database
furry 4 legs tail furry 4 legs tail
attributes
furry 4 legs tail …
Dog O O O
NN NN
class Fish X X O
Chimp O X X
…
sufficient attributes for one
to one mapping
Zero-shot Learning
• Representing each class by its attributes
Testing Find the class with the most
similar attributes
0 0 1
furry 4 legs tail attributes
furry 4 legs tail …
Dog O O O
NN
class Fish X X O
Chimp O X X
…
sufficient attributes for one
to one mapping
𝑓 ∗ and g ∗ can be NN.
Zero-shot Learning Training target:
𝑓 𝑥 𝑛 and 𝑔 𝑦 𝑛 as
close as possible
• Attribute embedding
x2 y1 (attribute
of chimp) y2 (attribute
x1 of dog)
2
𝑓 𝑥2 𝑔 𝑦
𝑓 𝑥1 𝑔 𝑦1
y3 (attribute of 𝑔 𝑦3 𝑓 𝑥3
x3
Alpaca)
Embedding Space
More about Zero-shot learning
• Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, Tom M.
Mitchell, “Zero-shot Learning with Semantic Output Codes”, NIPS
2009
• Zeynep Akata, Florent Perronnin, Zaid Harchaoui and Cordelia
Schmid, “Label-Embedding for Attribute-Based Classification”,
CVPR 2013
• Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff
Dean, Marc'Aurelio Ranzato, Tomas Mikolov, “DeViSE: A Deep
Visual-Semantic Embedding Model”, NIPS 2013
• Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram
Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, Jeffrey
Dean, “Zero-Shot Learning by Convex Combination of Semantic
Embeddings”, arXiv preprint 2013
• Subhashini Venugopalan, Lisa Anne Hendricks, Marcus
Rohrbach, Raymond Mooney, Trevor Darrell, Kate Saenko,
“Captioning Images with Diverse Objects”, arXiv preprint 2016
