Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
1 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Gaussian Mixture Models
Explained
From intuition to implementation
Oscar Contreras Carrasco Follow
Jun 3 · 12 min read
In the world of Machine Learning, we can distinguish two
main areas: Supervised and unsupervised learning. The
main difference between both lies in the nature of the data
as well as the approaches used to deal with it. Clustering is
an unsupervised learning problem where we intend to find
clusters of points in our dataset that share some common
characteristics. Let’s suppose we have a dataset that looks
like this:
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
2 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
. . .
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
3 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
De�initions
A Gaussian Mixture is a function that is comprised of several
Gaussians, each identified by k ∈ {1,…, K}, where K is the
number of clusters of our dataset. Each Gaussian k in the
mixture is comprised of the following parameters:
A mean μ that defines its centre.
A covariance Σ that defines its width. This would be
equivalent to the dimensions of an ellipsoid in a
multivariate scenario.
A mixing probability π that defines how big or small the
Gaussian function will be.
Let us now illustrate these parameters graphically:
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
Here, we can see that there are three Gaussian functions,
4 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
. . .
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
5 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Initial derivations
We are now going to introduce some additional notation.
Just a word of warning. Math is coming on! Don’t worry. I’ll
try to keep the notation as clean as possible for better
understanding of the derivations. First, let’s suppose we
want to know what is the probability that a data point xn
comes from Gaussian k. We can express this as:
Which reads “given a data point x, what is the probability it
came from Gaussian k?” In this case, z is a latent variable that
takes only two possible values. It is one when x came from
Gaussian k, and zero otherwise. Actually, we don’t get to see
this z variable in reality, but knowing its probability of
occurrence will be useful in helping us determine the
Gaussian mixture parameters, as we discuss later.
Likewise, we can state the following:
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
Which means that the overall probability of observing a
6 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
. . .
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
7 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Expectation — Maximization algorithm
Well, at this point we have derived some expressions for the
probabilities that we will find useful in determining the
parameters of our model. However, in the past section we
could see that simply evaluating (3) to find such parameters
would prove to be very hard. Fortunately, there is an
iterative method we can use to achieve this purpose. It is
called the Expectation — Maximization, or simply EM
algorithm. It is widely used for optimization problems where
the objective function has complexities such as the one we’ve
just encountered for the GMM case.
Let the parameters of our model be
Let us now define the steps that the general EM algorithm
will follow¹.
Step 1: Initialise θ accordingly. For instance, we can use the
Get one
results moreby
obtained story in your
a previous member
K-Means run as a good
preview when
starting point youalgorithm.
for our sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
Step 2 (Expectation step): Evaluate account? Sign in
8 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
. . .
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
9 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Implementation in Python
Just as a side note, the full implementation is available as a
Jupyter notebook at https://bit.ly/2MpiZp4
I have used the Iris dataset for this exercise, mainly for
simplicity and fast training. From our previous derivations,
we stated that the EM algorithm follows an iterative
approach to find the parameters of a Gaussian Mixture
Model. Our first step was to initialise our parameters. In this
case, we can use the values of K-means to suit this purpose.
The Python code for this would look like:
1 def initialize_clusters(X, n_clusters):
2 clusters = []
3 idx = np.arange(X.shape[0])
4
5 kmeans = KMeans().fit(X)
6 mu_k = kmeans.cluster_centers_
7
8 for i in range(n_clusters):
9 clusters.append({
10 'pi_k': 1.0 / n_clusters,
11 'mu_k': mu_k[i],
12 'cov_k': np.identity(X.shape[1], dtype=np.float64)
13 })
14
Get one more story in your member
Next, we execute the expectation step. Here we calculate
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
10 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
. . .
Final remarks
Gaussian Mixture Models are a very powerful tool and are
widely used in diverse tasks that involve data clustering. I
hope you found this post useful! Feel free to approach with
questions or comments. I would also highly encourage you to
try the derivations yourself as well as look further into the
code. I look forward to creating more material like this soon.
Enjoy!
. . .
[1] Bishop, Christopher M. Pattern Recognition and Machine
Learning (2006) Springer-Verlag Berlin, Heidelberg.
[2] Murphy, Kevin P. Machine Learning: A Probabilistic
Perspective (2012) MIT Press, Cambridge, Mass,
Machine Learning Gaussian Mixture Model Gmm Clustering
Get one
Towards Datamore
Science story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
11 of 12 11/7/19, 2:24 PM
Gaussian Mixture Models Explained - Towards ... https://towardsdatascience.com/gaussian-mixtu...
Discover Medium Make Medium Become a member
Welcome to a place where
yours Get unlimited access to
words matter. On Follow all the topics you the best stories on
Medium, smart voices and care about, and we’ll Medium — and support
original ideas take center deliver the best stories for writers while you’re at it.
stage - with no ads in you to your homepage and Just $5/month. Upgrade
sight. Watch inbox. Explore
About Help Legal
Get one more story in your member
preview when you sign up. It’s free.
Already have an
Sign up with Google Sign up with Facebook
account? Sign in
12 of 12 11/7/19, 2:24 PM
