Neural Models for NLP
Lecture 15
COMS 4705, Spring 2020
Yassine Benajiba
Columbia University
One more thing about embeddings
No learning
No learning
If you don’t want to use any additional learning, you could just stop here and take the average of the
embeddings as the representation of the sentence. What are the pros and cons of such an approach?
What if you wanted to weight the words without any additional learning, what could you use?
doc2vec
Doc2vec/paragraph2vec
https://medium.com/district-data-labs/forward-propagation-building-a-skip-gram-net-from-the-ground-up-9578814b221
At the word level, Skip gram takes a word as an input and predicts its context.
We can do few changes and get a document representation.
p’: Matrix to increase # dimensions
Doc2vec/paragraph2vec This is what we throw away at the end
p: Matrix to reduce # dimensions
One row per word
This is what we save at the end as the word embedding matrix
Input word one hot encoding
One hot encoded
context words
p’: Matrix to increase # dimensions
Doc2vec/paragraph2vec This is what we throw away at the end
p: Matrix to reduce # dimensions
One row per word doc
This is what we save at the end as the word doc embedding matrix
vd
d(t)
Input word doc one hot encoding
We should call it d(t) instead
One hot encoded
context words
occurring in the
document
p’: Matrix to increase # dimensions
Doc2vec/paragraph2vec This is what we throw away at the end
p: Matrix to reduce # dimensions
One row per word doc
This is what we save at the end as the word doc embedding matrix
vd
d(t)
Input word doc one hot encoding
We should call it d(t) instead
What happens for documents that were not in the One hot encoded
training? context words
occurring in the
Add new column to d(t) and new parameter row to ‘p’, document
freeze all old parameters and train for few iterations.
p’: Matrix to increase # dimensions
Doc2vec/paragraph2vec This is what we throw away at the end
p: Matrix to reduce # dimensions
One row per word doc
This is what we save at the end as the word doc embedding matrix
vd
d(t)
Input word doc one hot encoding
We should call it d(t) instead
Approach doesn’t allow for composition, i.e. does not build
document representation from words.
One hot encoded
context words
Nice for document similarity but not for applications like
occurring in the
machine translation, question answering, sentiment, etc
document
Let’s take a look at the ones that attempt to do so.
dense nets
dense nets
dense nets
Deep Averaging Networks (DANs)
https://cs.umd.edu/~miyyer/pubs/2015_acl_dan.pdf
dense nets
Deep Averaging Networks (DANs)
dense nets
Deep Averaging Networks (DANs)
dense nets
Deep Averaging Networks (DANs)
dense nets
Deep Averaging Networks (DANs)
RAND -> randomly initialized embeddings
NBOW -> avg + softmax
ROOT -> sentiment at the sentence level
dense nets
Deep Averaging Networks (DANs)
RAND -> randomly initialized embeddings
NBOW -> avg + softmax
ROOT -> sentiment at the sentence level
dense nets
Deep Averaging Networks (DANs)
dense nets
Deep Averaging Networks (DANs)
dense nets
Deep Averaging Networks (DANs)
Simple
Fast
But what are we missing here?
Primarily order of words. It could beat
RecNN but not CNN!
convolutional nets
CNNs
intuition
CNNs
feature maps
• A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification.
Zhang Y. and Wallace B.
https://arxiv.org/pdf/1510.03820v2.pdf
CNNs
the convolution operation
Wikipedia
CNNs
the convolution operation
1D
Wikipedia
Input
kernel Output
2D
* =
CNNs
the convolution operation
A (input)
B (kernel) C (output)
* =
C 𝑚, 𝑛 = 𝐴 ∗ 𝐵 𝑚, 𝑛 = σ𝑗 σ𝑘 𝐵 𝑗, 𝑘 . 𝐴[𝑚 + 𝑗, 𝑛 + 𝑘]
CNNs
- Shared parameters
- Sparse neurons
CNNs Retain only the max values.
Why does this make sense?
https://arxiv.org/pdf/1510.03820v2.pdf
Convolutional neural networks
https://github.com/bwallace/CNN-for-text-classification/blob/master/CNN_text.py
CNNs
Now we have something that can take
word order into consideration but we still
have two problems:
- Length of sentence
- Long distance dependencies
.. Let’s talk about recurrent neural
networks
https://arxiv.org/pdf/1510.03820v2.pdf
recurrent nets
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
How can we do this with words
instead of chars?
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
sequential
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
Shared params
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
generative
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
Last hidden state is sequence
representation
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
Can be used to classify either each element
of the sequence or whole sequence
LMs too
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
RNNs
Vanilla
The Unreasonable Effectiveness of Recurrent Neural Networks
Training:
Algorithm: BTTBP
Issues: vanishing gradients
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Gated RNNs: LSTMs
Gated RNNs: LSTMs
Ct-1 Ct
ht-1 ht
xt
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Gated RNNs: LSTMs
Ct-1 Ct
ht-1 ht
xt
Gated RNNs: LSTMs
Ct-1 Ct
ht-1 ht
xt
Gated RNNs: LSTMs
Ct-1 Ct
ht-1 ht
xt Given an NLP task the LSTM will pick what information to remember and what information is
irrelevant to build a sentence level rep: h(n) where ‘n’ is the length of the sentence
Gated RNNs: LSTMs
Ct-1 Ct
Works much better ..
What problems does it solve?:
- Vanishing gradient
ht-1 ht - Longer memory
xt
Gated RNNs: LSTMs
Gated RNNs: LSTMs
BiLSTM means bidirectional
LSTM, basically two LSTMs one
starting from the left and the
other from the right. At each
word we concatenate the
content of the hidden layers
Gated RNNs: LSTMs
Let’s discuss some
limitations of LSTMs:
- Speed
- Accuracy of
representation
Where do you think
is the state of art
today?
Gated RNNs: LSTMs
http://nlpprogress.com/
Gated RNNs: LSTMs
http://nlpprogress.com/
Other architectures
ADANs
one-shot learning
Back to the intro
Back to the intro
Back to the intro
Back to the intro
