Non-negative matrix
factorization (NMF)
UNSUPERVISED LEARNING IN PYTHON
Benjamin Wilson
Director of Research at lateral.io
Non-negative matrix factorization
NMF = "non-negative matrix factorization"
Dimension reduction technique
NMF models are interpretable (unlike PCA)
Easy to interpret means easy to explain!
However, all sample features must be non-negative (>= 0)
UNSUPERVISED LEARNING IN PYTHON
Interpretable parts
NMF expresses documents as combinations of topics (or
"themes")
UNSUPERVISED LEARNING IN PYTHON
Interpretable parts
NMF expresses images as combinations of pa erns
UNSUPERVISED LEARNING IN PYTHON
Using scikit-learn NMF
Follows fit() / transform() pa ern
Must specify number of components e.g.
NMF(n_components=2)
Works with NumPy arrays and with csr_matrix
UNSUPERVISED LEARNING IN PYTHON
Example word-frequency array
Word frequency array, 4 words, many documents
Measure presence of words in each document using "tf-idf"
"tf" = frequency of word in document
"idf" reduces in uence of frequent words
UNSUPERVISED LEARNING IN PYTHON
Example usage of NMF
samples is the word-frequency array
from sklearn.decomposition import NMF
model = NMF(n_components=2)
model.fit(samples)
NMF(alpha=0.0, ... )
nmf_features = model.transform(samples)
UNSUPERVISED LEARNING IN PYTHON
NMF components
NMF has components
... just like PCA has principal components
Dimension of components = dimension of samples
Entries are non-negative
print(model.components_)
[[ 0.01 0. 2.13 0.54]
[ 0.99 1.47 0. 0.5 ]]
UNSUPERVISED LEARNING IN PYTHON
NMF features
NMF feature values are non-negative
Can be used to reconstruct the samples
... combine feature values with components
print(nmf_features)
[[ 0. 0.2 ]
[ 0.19 0. ]
...
[ 0.15 0.12]]
UNSUPERVISED LEARNING IN PYTHON
Reconstruction of a sample
print(samples[i,:])
[ 0.12 0.18 0.32 0.14]
print(nmf_features[i,:])
[ 0.15 0.12]
UNSUPERVISED LEARNING IN PYTHON
Sample reconstruction
Multiply components by feature values, and add up
Can also be expressed as a product of matrices
This is the "Matrix Factorization" in "NMF"
UNSUPERVISED LEARNING IN PYTHON
NMF fits to non-negative data only
Word frequencies in each document
Images encoded as arrays
Audio spectrograms
Purchase histories on e-commerce sites
... and many more!
UNSUPERVISED LEARNING IN PYTHON
Let's practice!
UNSUPERVISED LEARNING IN PYTHON
NMF learns
interpretable parts
UNSUPERVISED LEARNING IN PYTHON
Benjamin Wilson
Director of Research at lateral.io
Example: NMF learns interpretable parts
Word-frequency array articles (tf-idf)
20,000 scienti c articles (rows)
800 words (columns)
UNSUPERVISED LEARNING IN PYTHON
Applying NMF to the articles
print(articles.shape)
(20000, 800)
from sklearn.decomposition import NMF
nmf = NMF(n_components=10)
nmf.fit(articles)
NMF(alpha=0.0, ... )
print(nmf.components_.shape)
(10, 800)
UNSUPERVISED LEARNING IN PYTHON
NMF components are topics
UNSUPERVISED LEARNING IN PYTHON
NMF components are topics
UNSUPERVISED LEARNING IN PYTHON
NMF components are topics
UNSUPERVISED LEARNING IN PYTHON
NMF components are topics
UNSUPERVISED LEARNING IN PYTHON
NMF components
For documents:
NMF components represent topics
NMF features combine topics into documents
For images, NMF components are parts of images
UNSUPERVISED LEARNING IN PYTHON
Grayscale images
"Grayscale" image = no colors, only shades of gray
Measure pixel brightness
Represent with value between 0 and 1 (0 is black)
Convert to 2D array
UNSUPERVISED LEARNING IN PYTHON
Grayscale image example
An 8x8 grayscale image of the moon, wri en as an array
UNSUPERVISED LEARNING IN PYTHON
Grayscale images as flat arrays
Enumerate the entries
Row-by-row
From le to right, top to bo om
UNSUPERVISED LEARNING IN PYTHON
Grayscale images as flat arrays
Enumerate the entries
Row-by-row
From le to right, top to bo om
UNSUPERVISED LEARNING IN PYTHON
Encoding a collection of images
Collection of images of the same size
Encode as 2D array
Each row corresponds to an image
Each column corresponds to a pixel
... can apply NMF!
UNSUPERVISED LEARNING IN PYTHON
Visualizing samples
print(sample)
[ 0. 1. 0.5 1. 0. 1. ]
bitmap = sample.reshape((2, 3))
print(bitmap)
[[ 0. 1. 0.5]
[ 1. 0. 1. ]]
from matplotlib import pyplot as plt
plt.imshow(bitmap, cmap='gray', interpolation='nearest')
plt.show()
UNSUPERVISED LEARNING IN PYTHON
Let's practice!
UNSUPERVISED LEARNING IN PYTHON
Building
recommender
systems using NMF
UNSUPERVISED LEARNING IN PYTHON
Benjamin Wilson
Director of Research at lateral.io
Finding similar articles
Engineer at a large online newspaper
Task: recommend articles similar to article being read by
customer
Similar articles should have similar topics
UNSUPERVISED LEARNING IN PYTHON
Strategy
Apply NMF to the word-frequency array
NMF feature values describe the topics
... so similar documents have similar NMF feature values
Compare NMF feature values?
UNSUPERVISED LEARNING IN PYTHON
Apply NMF to the word-frequency array
articles is a word frequency array
from sklearn.decomposition import NMF
nmf = NMF(n_components=6)
nmf_features = nmf.fit_transform(articles)
UNSUPERVISED LEARNING IN PYTHON
Strategy
Apply NMF to the word-frequency array
NMF feature values describe the topics
... so similar documents have similar NMF feature values
Compare NMF feature values?
UNSUPERVISED LEARNING IN PYTHON
Versions of articles
Di erent versions of the same document have same topic
proportions
... exact feature values may be di erent!
UNSUPERVISED LEARNING IN PYTHON
Versions of articles
Di erent versions of the same document have same topic
proportions
... exact feature values may be di erent!
E.g. because one version uses many meaningless words
UNSUPERVISED LEARNING IN PYTHON
Versions of articles
Di erent versions of the same document have same topic
proportions
... exact feature values may be di erent!
E.g. because one version uses many meaningless words
But all versions lie on the same line through the origin
UNSUPERVISED LEARNING IN PYTHON
Cosine similarity
Uses the angle between the lines
Higher values means more similar
Maximum value is 1, when angle is 0 degrees
UNSUPERVISED LEARNING IN PYTHON
Calculating the cosine similarities
from sklearn.preprocessing import normalize
norm_features = normalize(nmf_features)
# if has index 23
current_article = norm_features[23,:]
similarities = norm_features.dot(current_article)
print(similarities)
[ 0.7150569 0.26349967 ..., 0.20323616 0.05047817]
UNSUPERVISED LEARNING IN PYTHON
DataFrames and labels
Label similarities with the article titles, using a DataFrame
Titles given as a list: titles
import pandas as pd
norm_features = normalize(nmf_features)
df = pd.DataFrame(norm_features, index=titles)
current_article = df.loc['Dog bites man']
similarities = df.dot(current_article)
UNSUPERVISED LEARNING IN PYTHON
DataFrames and labels
print(similarities.nlargest())
Dog bites man 1.000000
Hound mauls cat 0.979946
Pets go wild! 0.979708
Dachshunds are dangerous 0.949641
Our streets are no longer safe 0.900474
dtype: float64
UNSUPERVISED LEARNING IN PYTHON
Let's practice!
UNSUPERVISED LEARNING IN PYTHON
Final thoughts
UNSUPERVISED LEARNING IN PYTHON
Benjamin Wilson
Director of Research at lateral.io
Congratulations!
UNSUPERVISED LEARNING IN PYTHON
