Boltzmann Machine
• each nodes connects to every
other node in the network
• useful for factory safety when
an unusual state occurs to
alert the workers (the state
might not have occurred
previously).
Restricted Boltzmann Machine (RBM)
• RBMs
• models probability
• feature extraction
• Dimensionality reduction
• perform unsupervised pretraining of the rst layer based on the input data
• They are feed-forward networks in which data is fed through them in one
direction with two biases rather than one bias as in traditional back
propagation feed-forward networks.
fi
Restricted Boltzmann Machine (RBM)
• The “restricted” part of the name
“Restricted Boltzmann Machines” means
that connections between nodes of the
same layer are prohibited (e.g., there are
no visible-visible or hidden-hidden
connections along which signal passes)
• " A network of symmetrically connected,
neuron-like units that make stochastic
decisions about whether to be on or o ."
• RBM is a Stochastic Neural Network
which means that each neuron will have
random behavior when activated.
ff
Restricted Boltzmann Machine (RBM)
• Parts of RBM
• Visible units
• Hidden units
• Weights
• Visible bias units
• Hidden bias units
Restricted Boltzmann Machine (RBM)
• Visible and hidden layers
• every single node of the input (visible) layer is connected by weights to every
single node of the hidden layer, but no two nodes of the same layer are connected
• The second layer is known as the “hidden” layer. Hidden units are feature
detectors, learning features from the input data
• Each layer has a bias unit with its state always set to on.
• Each node performs computation based on the input to the node and outputs a
result based on a stochastic decision whether or not to transmit data through an
activation.
• The activation computation is based on weights on the connections and the input
values. The initial weights are randomly generated.
Restricted Boltzmann Machine (RBM)
• Connections and weights
• All connections are visible-hidden; none are visible-visible or hidden-
hidden.
• Those circles, or nodes, act like human neurons. They are decision
units.
• They make decisions about whether to be on or o through acts of
computation. “On” means that they pass a signal further through the
net; “o ” means that they don’t.
• ON is valuable and OFF signals may be due to a noisy input.
ff
ff
Restricted Boltzmann Machine (RBM)
• Hidden bias and visible bias
• This is what makes RBMs di erent from autoencoders.
• The hidden bias RBM produce the activation on the forward pass.
• The visible bias helps RBM to reconstruct the input during a backward
pass.
• The reconstructed input is always di erent from the actual input.
• As there are no connections among the visible units and therefore, no
way of transferring information among themselves.
ff
ff
Training RBM
• Forward Pass
• There can be multiple inputs. All inputs are multiplied by the weights and then added to the
bias. Then the result is passed through sigmoid activation function.
• Reconstruction Phase
• It is like the forward pass but in the opposite direction.
• h1: hidden layer vector, v0: input vector, W: weight matrix, a: hidden layer bias vector, S:
sigmoid activation function, b: visible layer bias
• The di erence (v0-v1) can be considered as the reconstruction error that we need to reduce
in subsequent steps of the training process. The weights are adjusted in each iteration to
minimize this error.
ff
Training RBM
Training RBM
• Suppose we have two normal
distributions,
• one from the input data (p(x))
• one from the reconstructed input (q(x)).
• The di erence between these two
distributions is the error and the goal is
to minimize it.
• KL-divergence (Kullback-Leibler
divergence) measures the non-
overlapping areas under the two
graphs.
ff
Case Study on RBM
• MNIST (Mixed National Institute of
Standards and Technology)
• 60000 records and 10000 test
records
• Cross entropy
• Cross means comparison
between two distributions
• entropy means uncertainty
(where the points fall on a normal
distribution curve if the variance is
wide spread).
Case Study on RBM
Applications of RBM
• Recommender System:
• Amazon product suggestions
• Net ix movie recommendations
• Handwritten digit recognition
• Criminal evidence,
• O ce computerization,
• Check (Cheque) veri cation,
• Data entry applications
• Vehicle number plate identi cation
ffi
fl
fi
fi
