Autoencoder

Autoencoders are self-supervised systems whose training goal is to compress (or encode)
input data through dimensionality reduction and then accurately reconstruct (or decode) their
original input by using that compressed representation.

The function of an autoencoder is to effectively extract the data’s most salient information—
its latent variables—and discard irrelevant noise.

The encoder network passes input data from the training data set through a "bottleneck"
before it reaches the decoder.

The decoder network, in turn, is then responsible for reconstructing the original input by
using only the vector of latent variables.
Autoencoder Structure

The encoder extracts latent variables of input data x and outputs them in the form of a vector
representing latent space z. each subsequent layer of the encoder contains progressively
fewer nodes than the previous layer; as data traverses each encoder layer, it’s compressed
through the process of “squeezing” itself into fewer dimensions.

The bottleneck, or "code," is both the output layer of the encoder network and the input layer
of the decoder network. It contains the latent space: the fully compressed, lower-dimensional
embedding of the input data.

The decoder uses that latent representation to reconstruct the original input by essentially
reversing the encoder: in a typical decoder architecture, each subsequent layer contains a
progressively larger number of active nodes.
Encoder Network Architecture
Decoder Network Architecture
Types of Autoencoder
❖ Sparse autoencoders: Suitable for feature extraction, dimensionality reduction, anomaly detection and
transfer learning. They use techniques to encourage the neural network to use only a subset of the
intermediate neurons. This surplus of unused neurons gives them the flexibility to identify and learn a more
efficient representation of data.
❖ Denoising autoencoders. Learn ways to reconstruct the original data from a noisy data stream. They're often
used to clean up low-light images, recognize speech and preprocess IoT data.
❖ Contractive autoencoders. Specialize in learning a representation that can adapt to small changes in the
input data. This helps them better adapt to unseen data. Researchers highlight the most salient features in the
data set responsible for results to improve the interpretability of neural network models.
❖ Undercomplete autoencoder. Specialize in reducing or minimizing the completeness of input data. Useful in
tasks such as data compression and anomaly detection.
❖ Convolutional autoencoders: Ability to encode and decode complex data and to minimize the reconstruction
error between input and output data elements makes these models well suited for image reconstruction,
denoising, and learning.
❖ Conditional autoencoders: VAEs designed to improve generative capabilities by selecting variations based
on conditional information such as attributes, context or labels. Conditional VAEs are excellent choices for
visualization and image generation.
Variational Autoencoder (VAE)
Introduction
❖ Variational Autoencoders (VAEs) are a type of artificial neural network architecture
that combines the power of autoencoders with probabilistic methods.
❖ They utilize an architecture that comprises an encoder-decoder structure.
❖ The encoder transforms input data into a latent form, and the decoder aims to
reconstruct the original data based on this latent representation.
❖ A variational autoencoder (VAE) is one of several generative models that use deep
learning to generate new content, detect anomalies and remove noise.
❖ VAEs first appeared in 2013, same time Generative AI algorithms, such as generative
adversarial networks (GANs) and diffusion models, but earlier than large language
models built on BERT, the GPT family and the Pathways Language Model.
Architecture of VAE

❖ VAE typically has two major components: An encoder connection and a decoder connection.
An encoder network transforms The input data into a low-dimensional secret space, often
called a “secret code”.
❖ ully connected or convolutional neural networks, can be investigated for implementing
encoder networks. The architecture chosen is based on the characteristics of the data.
❖ The encoder network produces essential parameters, such as the mean and variance of a
Gaussian distribution, necessary for sampling and generating the latent code
VAE works
VAE is the Gaussian mixture model (GMM), code is replaced by a continuous variable z, and z
follow standard normal distribution N(0,1).

Two variables μ and σ, which respectively determine the mean and standard deviation of the
Gaussian distribution corresponding to z, and then the accumulation of all Gaussian distributions
in the integration domain becomes the original distribution P(x):
Application of variational autoencoders

❖ VAEs have three fundamental purposes: create new data, identify anomalies in data,
and remove noisy or unwanted data. VAEs well suited for numerous powerful
applications, such as the following:
➢ Text creation. VAEs can generate new text about a topic in a desired style. However, the
new text is limited to the scope of training text and the desired style.
➢ Image creation. VAEs can generate new images, though the images are limited by what's
available in the training data set.
➢ Video creation. VAEs can produce new video sequences
➢ Language processing. VAEs can recognize and understand complex relationships between
data elements.
➢ Anomaly detection. VAEs can process enormous volumes of data and specialize in time
series or sequential data processing.
➢ Synthetic data creation. Software developers use VAEs to generate synthetic data sets for
software development and testing.