Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Boltzmann Machine
A Brief Introduction
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar
Arunabha Saha
Calcutta University
October 23, 2013
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
1 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Stochastic Hopfield Nets with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
Boltzmann Machine
Learning Algorithm for Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
2 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Hopfield Network
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Figure : Two dimensional representation of motion in state space
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
3 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline
A Hopfield Net is composed of binary threshold units with
recurrent connections between them.
Hopfield Net
Recurrent networks of non-linear units are hard to analyze,
since they can behave in many different ways -
Boltzmann Machine
Settle to a stable state.
Oscillate
Follow chaotic trajectory.
Ritajit Majumdar Arunabha Saha (CU)
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline
A Hopfield Net is composed of binary threshold units with
recurrent connections between them.
Hopfield Net
Recurrent networks of non-linear units are hard to analyze,
since they can behave in many different ways -
Boltzmann Machine
Settle to a stable state.
Oscillate
Follow chaotic trajectory.
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
John Hopfield introduced a global energy function for
network with symmetric connections.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline
A Hopfield Net is composed of binary threshold units with
recurrent connections between them.
Hopfield Net
Recurrent networks of non-linear units are hard to analyze,
since they can behave in many different ways -
Boltzmann Machine
Settle to a stable state.
Oscillate
Follow chaotic trajectory.
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
John Hopfield introduced a global energy function for
network with symmetric connections.
1
Each binary configuration of the whole network has an
Energy.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline
A Hopfield Net is composed of binary threshold units with
recurrent connections between them.
Hopfield Net
Recurrent networks of non-linear units are hard to analyze,
since they can behave in many different ways -
Boltzmann Machine
Settle to a stable state.
Oscillate
Follow chaotic trajectory.
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
John Hopfield introduced a global energy function for
network with symmetric connections.
1
Each binary configuration of the whole network has an
Energy.
The binary threshold decision rule causes the network to
settle to a minimum of this energy function.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline
The global energy is defined as X
X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th
where bi is the bias of the i unit, s is 0 or 1 depending on
whether the unit is turned off or on respectively. And wij is the
weight of the connection between units i and j.
1 For
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
Reference
Bipolar Inputs the states will be 1 and 1 respectively.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
5 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline
The global energy is defined as X
X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th
Stochastic Hopfield Nets
with Hidden Units
where bi is the bias of the i unit, s is 0 or 1 depending on
whether the unit is turned off or on respectively. And wij is the
weight of the connection between units i and j.
Learning Algorithm for
Boltzmann Machine
Reference
From this energy function, each unit computes locally how
changing their state will affect the global energy.
1 For
Bipolar Inputs the states will be 1 and 1 respectively.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
5 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline
The global energy is defined as X
X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for
Boltzmann Machine
where bi is the bias of the i unit, s is 0 or 1 depending on
whether the unit is turned off or on respectively. And wij is the
weight of the connection between units i and j.
Reference
From this energy function, each unit computes locally how
changing their state will affect the global energy.
The energy gap is defined as 4 Ei = E (si = 0) E (si = 1) = bi +
sj wij
(2)
1 For
Bipolar Inputs the states will be 1 and 1 respectively.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
5 / 28
Settling to an Energy Minima
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
The net is initially in a random
state i.e., the units are on or off
randomly. The binary threshold
decision rule updates units one
at a time in a random order.
Update each unit to
whichever of its two states
minimizes the global
energy.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Use binary threshold units,
i.e., the states can be
either 0 or 1.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
6 / 28
Settling to an Energy Minima
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
The net is initially in a random
state i.e., the units are on or off
randomly. The binary threshold
decision rule updates units one
at a time in a random order.
Update each unit to
whichever of its two states
minimizes the global
energy.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Use binary threshold units,
i.e., the states can be
either 0 or 1.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
7 / 28
Settling to an Energy Minima
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
The net is initially in a random
state i.e., the units are on or off
randomly. The binary threshold
decision rule updates units one
at a time in a random order.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Update each unit to
whichever of its two states
minimizes the global
energy.
Use binary threshold units,
i.e., the states can be
either 0 or 1.
Ritajit Majumdar Arunabha Saha (CU)
-E = goodness = 4
Boltzmann Machine
October 23, 2013
8 / 28
Using this type of Computation as memory
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Hopfield proposed that memories could be energy minima
of a neural net.
I
The binary threshold decision rule can be used to clean
up incomplete or corrupted memory.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
October 23, 2013
9 / 28
Using this type of Computation as memory
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Hopfield proposed that memories could be energy minima
of a neural net.
I
The binary threshold decision rule can be used to clean
up incomplete or corrupted memory.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Using energy minima to represent memories gives a
content-addressable memory.
I
I
An item can be accessed just by knowing parts of it.
It is robust against hardware damage.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
9 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Stochastic Hopfield Nets with Hidden
Units
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
10 / 28
Hopfield Net with Hidden Units
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Why add hidden units?
Stochastic Hopfield Nets
with Hidden Units
In Hopfield Net, there is no hidden layer of units. By adding
hidden layers, the attention can be shifted from just storing
memories to various types of interpretations of the inputs.
Boltzmann Machine
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
October 23, 2013
11 / 28
Hopfield Net with Hidden Units
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Why add hidden units?
Stochastic Hopfield Nets
with Hidden Units
In Hopfield Net, there is no hidden layer of units. By adding
hidden layers, the attention can be shifted from just storing
memories to various types of interpretations of the inputs.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Use the net to construct
interpretations of sensory
input.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
11 / 28
Hopfield Net with Hidden Units
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Why add hidden units?
Stochastic Hopfield Nets
with Hidden Units
In Hopfield Net, there is no hidden layer of units. By adding
hidden layers, the attention can be shifted from just storing
memories to various types of interpretations of the inputs.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Use the net to construct
interpretations of sensory
input.
The input is represented
by the visible units.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
11 / 28
Hopfield Net with Hidden Units
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Why add hidden units?
Stochastic Hopfield Nets
with Hidden Units
In Hopfield Net, there is no hidden layer of units. By adding
hidden layers, the attention can be shifted from just storing
memories to various types of interpretations of the inputs.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Use the net to construct
interpretations of sensory
input.
The input is represented
by the visible units.
The interpretation is
represented by the states
of the hidden units.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
11 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Hence it is impossible to escape
from a local minima.
Learning Algorithm for
Boltzmann Machine
Ritajit Majumdar Arunabha Saha (CU)
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Hence it is impossible to escape
from a local minima.
Learning Algorithm for
Boltzmann Machine
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Reference
Solution - Use random noise to
escape from poor, shallow
minima.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Hence it is impossible to escape
from a local minima.
Learning Algorithm for
Boltzmann Machine
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Reference
Solution - Use random noise to
escape from poor, shallow
minima.
I
Start with a lot of noise to
escape the energy barriers of
poor local minima.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Hence it is impossible to escape
from a local minima.
Learning Algorithm for
Boltzmann Machine
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Reference
Solution - Use random noise to
escape from poor, shallow
minima.
I
Start with a lot of noise to
escape the energy barriers of
poor local minima.
Slowly reduce the noise so
that the system ends up in a
deep minima.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision
rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
Hence it is impossible to escape
from a local minima.
Learning Algorithm for
Boltzmann Machine
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Reference
Solution - Use random noise to
escape from poor, shallow
minima.
I
Start with a lot of noise to
escape the energy barriers of
poor local minima.
Slowly reduce the noise so
that the system ends up in a
deep minima.
This process is called
Simulated Annealing.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
1
Hopfield Net
Replace the binary threshold units by binary stochastic
units that make biased random decisions.
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
1
Hopfield Net
Replace the binary threshold units by binary stochastic
units that make biased random decisions.
Stochastic Hopfield Nets
with Hidden Units
The temperature controls the amount of noise.
Learning Algorithm for
Boltzmann Machine
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net
Replace the binary threshold units by binary stochastic
units that make biased random decisions.
Stochastic Hopfield Nets
with Hidden Units
The temperature controls the amount of noise.
Learning Algorithm for
Boltzmann Machine
Unit i then turns on with the probability given by the
logistic function -
Reference
prob(si = 1) =
Ritajit Majumdar Arunabha Saha (CU)
1
1 + e
4Ei
T
Boltzmann Machine
Boltzmann Machine
(3)
October 23, 2013
13 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net
Replace the binary threshold units by binary stochastic
units that make biased random decisions.
Stochastic Hopfield Nets
with Hidden Units
The temperature controls the amount of noise.
Learning Algorithm for
Boltzmann Machine
Unit i then turns on with the probability given by the
logistic function -
Reference
prob(si = 1) =
T =0
T
T =1
1
1 + e
4Ei
T
Boltzmann Machine
(3)
Deterministic (Hopfield Net)
Complete Chaos
Approaches Boltzmann Distribution
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
13 / 28
What is Boltzmann Machine?
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
A Hopfield Net consisting of Binary Stochastic Neuron with
hidden units is called Boltzmann Machine.
Reference
A Boltzmann Machine is a network of symmetrically
connected, neuron like units that make stochastic decisions
about whether to be on or off.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
14 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Boltzmann Machine
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
15 / 28
Boltzmann Machine
Modeling Binary Data
Ritajit Majumdar
Arunabha Saha
The objective of Boltzmann Machine is -
Outline
Hopfield Net
Modeling Binary Data
Stochastic Hopfield Nets
with Hidden Units
Given a training set of binary vectors, fit the model that will
assign a probability to every possible binary vector.
When unit i is given oppurtunity to update its state, it first
computes its total input zi ,
X
zi = bi +
sj wij
(4)
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Unit i turns on with probability prob(si = 1) =
1
1 + e zi
(5)
If the units are updated sequencially in random order, the
network will eventually reach a Boltzmann Distribution, also
called its stationary distribution.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
16 / 28
Boltzmann Machine
How Boltzmann Machine Generates Data
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
In Boltzmann Machine, everything is defined in terms of the
energies of joint configurations of the visibile (v) and hidden
(h) units.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
The energies of joint configurations are related to their
probabilities as p(v , h) e E (v ,h)
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
(6)
October 23, 2013
17 / 28
Probabilities in term of Energies
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
The probability of a joint
configuration over both
visibile and hidden units
depends on the energy of
that joint configuration
compared with the
energies of all other joint
configurations.
Ritajit Majumdar Arunabha Saha (CU)
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
18 / 28
Probabilities in term of Energies
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
The probability of a joint
configuration over both
visibile and hidden units
depends on the energy of
that joint configuration
compared with the
energies of all other joint
configurations.
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
The probability of a
configuration of the
visibile units is the sum of
the probabilities of all the
joint configurations that
contain it.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
18 / 28
Boltzmann Machine
An Example
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Figure : Credit: Geoffrey Hinton, Neural Networks for Machine
Learning
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
19 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Learning Algorithm
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
A surprising feature of this rule is that it uses only locally
available information. The change of weight depends only
on the behaviour of the two units it connects, even
though the change optimizes a global measure.
Reference
- Ackley, Hinton 1985
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
20 / 28
Boltzmann Machine
Goal of Learning
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Learning Algorithm for Boltzmann Machine is an unsupervised
learning algorithm. Unlike Backpropagation Algorithm, where
the training set consists of input vector and desired output, in
Boltzmann Machine only the input vector is provided.
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
We want to maximize the product of the probabilities the
Boltzmann Machine assigns to the binary vectors in
training set.
It is equivalent to maximizing the probability that we
would obtain exactly the N training cases.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
21 / 28
Why Learning could be difficult
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Consider a chain of hidden units with visibile units attached at
two ends -
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Training: We want the two visibile units to be in opposite
states.
Solution: The product of all the weights must be negative. If
all are positive, then turning on one unit will turn on the next
unit and eventually the two visibile units will be in same state.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
22 / 28
Why Learning could be difficult
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Consider a chain of hidden units with visibile units attached at
two ends -
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Difficulty: To modify w1 and w5, we need to know w3 (and
the weights of other hidden units too).
Because, if w3 is negative, then we need to modify w1 in a
different way than what we would do if w3 is positive.
So to change one weight in a right direction, we need to know
all the other weights.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
23 / 28
Boltzmann Machine
Learning Algorithm
Ritajit Majumdar
Arunabha Saha
Outline
Correlation
Hopfield Net
Everything that one weight needs to know of other weights and
the data is contained in the difference of two correlations logp(v )
= hsi sj iv hsi sj imodel
wij
(7)
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
where h.i is the expectation value.
The first term in R.H.S. denotes the expectation value of
product of states at equilibrium when the state vector (or
data) v is clamped on the visibile units.
The second term in R.H.S. denotes the expectation value
of product of states at equilibrium without any clamping.
So we can make the change in weight 4wij hsi sj iv hsi sj imodel
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
(8)
October 23, 2013
24 / 28
Boltzmann Machine
Why is it so?
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
We know the probability of a global configuration at
equilibrium is -
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
p(v , h) e E (v ,h)
Learning Algorithm for
Boltzmann Machine
So, the logarithm of probability is a linear function of the
energy.
Reference
And energy, on its own term, is a linear function of weights and
states.
P
P
E = si bi
si sj wij
i
i<j
Hence,
E
= si sj
wij
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
(9)
October 23, 2013
25 / 28
Boltzmann Machine
Why is it so?
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Differnetiating equation 1, we get E
wij
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
= si sj
Learning Algorithm for
Boltzmann Machine
Reference
The process of settling to equilibrium state propagates
information about the weights.
No need of backpropagation.
The following two stages are required -
I
I
4
The machine needs to settle to equilibrium with data.
The machine needs to settle to equilibrium without data.
However, in both cases the learning process is similar with
different boundary conditions.
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
26 / 28
Boltzmann Machine
Reference
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Geoffrey Hinton
Neural Networks for Machine Learning
www.coursera.org
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Geoffrey Hinton
http://www.scholarpedia.org/article/Boltzmann_
machine
Geoffrey Hinton, David Ackley
A learning Algorithm for Boltzmann Machine
Cognitive Science 9, 147-169 (1985)
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
27 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Stochastic Hopfield Nets
with Hidden Units
Boltzmann Machine
THANK YOU
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
October 23, 2013
28 / 28
