UNIT V

Autoencoders
5 and Generative Models

Syllabus autoencoders - Stochastic encoders and

Autoencoders : Undercomplete autoencoders - Regularized Models : Variational autoencoders
decoders - Learning with autoencoders; Deep Generative
Generative adversarial networks.

Contents
5.1 Autoencoders
5.2 Regularized Autoencoders
5.3 Stochastic Encoders and Decoders

5.4
Deep Generative Models
Generative Adversarial Networks
5.5
Variational Autoencoders
5.6
Answers
5.7 Two Marks Questions with
 Deep Learming 5-2 Autoencoders and Generative Models

5.1 Autoencoders
e Autoencoders play a fundamental role in unsupervised learning and in deen
architectures for transfer learning and other tasks. An autoencoder neural network is an
unsupervised learning algorithm that applies backpropagation, setting the target values
to be equal to the inputs.
An autoencoder is a special type of neural network that is trained to copy its input to its
output. Internally, it has a hidden layer h that describes a code used to represent the
input. The network may be viewed as consisting of two parts : An encoder
function
h=f(x) and a decoder that produces a reconstruction r=g (h).
For example, given an image of a handwritten digit, an
autoencoder first encodes the
image into a lower dimensional latent
representation, then decodes the latent
representation back to an image.
" An autoencoder learns to
compress the data while minimizing the
An autoencoder is unsupervised reconstruction error.
learning technique. It is an artificial neural network
used to learn data encodings of unlabeled data or
the task of representation
Properties of autoencoder: learning.
1. Data-specific :Autoencoders are only able to
what they have been trained on. meaningfully compress data similar to
Since they learn features
training data, they are different than a specific for the given
standard data
2. Lossy : The
output of the autoencoder will not be compression algorithm like gzip.
will be a close but exactly the same as the input, it
3.
degraded representation.
Unsupervised :
Autoencoders are considered an unsupervised
since they don't need
explicit labels to train on. learning technique
5.1.1 Architecture of
Fig. 5.1.1 shows
Autoencoder
" architecture of autoencoder.
Autoencoders are a specific type of
input to output. A
bottleneck feedforward neural networks trained to copy its
is imposed in
knowledge of the original the network to
and then the output is input. The input is represent a compressed
from thiscompressed
into a
linput.
atent-space reconstructed
representation which is a representation. lower-dimensional
The code is also
code
called as
compact "'summary" or
"compression of the

TECHNICAL an
eepLeeming
5-3 Autoencoders and Qenerathve Models
Input Ottput

Code

Decoder
Encoder

2 Encoder Decoder
Reconstructed
Original input
input Compressed
representation
autoencoder
Fig. 5.1.1 Architecture of
Encoder, code and decoder.
The encoder
of three components: input
Anautoencoder consists code, the decoder then reconstructs the
produces the
compresses the input and
encoded
only using this code. using afunction f. It then decodes the
values x,
Autoencoders encodes the input values identical to the input values.
functiong, to create output
Values f(X), using a techniques similar to
feedforward networks , training
autoencoder is a special case of gradient descent following gradients
" As
such as minibatch
feedforward neural network
by back-propagation can be used for training. neural networks. Code is
computed fully-connected feedforward
Both the encoder and decoder are
of user choice. The number of nodes
dimensionality
layer of an ANN with the training the autoencoder.
asingle hyperparameter that we set before
For
in thecode layer is a outputs under some constraints:
inputs to its
avtoencoder learns to copy its adding noise to the inputs.
An latent space or
dimensionality of the feature
example, limiting the input, encode it to identify latent
the
does autoencoders work ? We take representation to recreate the input. We
" How latent feature reconstruction error
representation. Decode the output. To reduce the
loss by comparing the input and
calculate the

up-thrust for knowledge

TECHNICAL PUBLICATIONS- an
Deep Leaming 5-4 Autoencoders and Generative Models

we back propagate and update the weights. Weight is updated based on how much they
are responsible for the error.
" There are four hyperparametersthat must be set before training the autoencoders. They
are as follows :
1. Code size : It is the number of nodes in the middle-layer. Smaller the size more is the
compression.
2. Number of layers: The autoencoder can be as deep as we like without considering
the input and output.
3. Number of nodes per layer: The number of nodes per layer decreases with each
subsequent layer of the encoder and increases back in the decoder. Also the layer
structure of decoder is symmetric to the encoder.
4. Loss function : Mean squared error or binary cross-entropy can be
used as loss
function. Cross-entropy is used if the input values are in the range [0,,1] else mean
squared error is used.
" The different types of autoencoders are as
follows :
1. Undercomplete autoencoders 2. Sparse
autoencoders
3. Contractive autoencoders
4. Denoising autoencoders
5. Variational autoencoders.

5.1.2 Use and Application of Autoencoder

Autoencoders are used for non linear dimensionality reduction.
hidden layer to a smaller dimension Encodes input in the
later decoded as output. Output layercompared the input dimension. Hidden layer is
to
has the same dimension as input.
reduces dimensionality of linear and
nonlinear data hence it is more powerfulAutoencoder
than PCA.
Autoencoders are used in recommendation engines. This
understand user preferences tO recommend uses deep encoders to
movies,
Autoencoders are used for feature extraction : books or items.
reconstruction error. In the process to reduce theAutoencoders
error, it
tries to minimize the
features present in the input. It learns some of important
the hidden layer. reconstructs the input from the encoded state
Encoding generates a new set of features present in
original features. which is a combination of the
in the input data. Encoding autoencoders helps to identify the latent
in
features presents
Image recognition : Stacked
multiple encoders stacked autoencoder are used for image recognition. We can use
together helps to learn different
features of an image.
 Models
peepLearning 5-5 Autoencoders and Generative

Two application of autoencoders are as follows :

1. Anomaly detection:
the data it commonly "sees" and
Aneural network trained with specific dataset learns
can also represent the difference between
represents the input dataset. This network
input and output and tell us when it "sees" unusual data.
used in such systems where it isdifficult to describe unusual or
Autoencoders can be
used for anomaly detection.
anomalous data. Undercomplete autoencoders are
dataset say "D", then it is supposed to
If autoencoder is trained on specific image
reconstruction loss.
reconstruct the image as it is with low
forimage which is not present in
task
But autoencoder cannot perform reconstruction be adapted by network for
any
attributes will not
the training dataset because latent
reconstruction loss for such image will be very
high.
unseen mage. As a result the easily identified as anomaly or unusual
threshold it can be
Soby setting appropriate
image.
good at powering anomaly detection systems.
" Due to this autoencoders are
of
2. Image denoising :
the proper information about the content
performed to obtain
Image denoising is can be used for this.
autoencoders
an image. Denoising in image, instead they extract the
not search for noise Then
Denoising autoencoders do their representations.
learning
data fed as input to them by
image from noisy
decompressing these representations
formed by image denoising
the noise free image is and highly accurate
perform efficient using
Denoising autoencoders can complex images that could not be denoised
denoising
and can be used for
traditional methods.

5.1.3 Undercomplete Autoencoders called

less than the input dimension is
is
whose code dimension
An autoencoder
can use to
undercomplete. unsupervised neural network that we
an image and
Undercomplete autoencoders is is done by taking in an
input data. It
version of the
generate a compressed reconstructing the image from its
predict the same image as output, thus
trying to
compressed bottleneck region.

knowledge
TECHNICAL PUBLCATIONS- an up-thrust for
 5-6
Autoencoders and Gonerative Mocdols
Deep Learning
generating a latent space or bottleneck.
The primary use for autoencoders like these is
can be casily decompressed
which forms a compressed substitute of the input data and
back with the help of the network when needed.
" Fig. 5.1.2shows undercomplete autoencoders. Output layer
Input layer Hidden layers

Decoder

autoencoders
Fig. 5.1.2 Undercomplete
autoencoder has no explicit regular1zation term - we simply train our
An undercomplete that the model
according to the reconstruction loss. Thus, our only way to ensure
model
data.
is not memorizing the input most
undercomplete representation forces the autoencoder to capture the
Learning an simply as
features of the training data. The learning process is described
salient function penalizing, g (f(x)
L(x, g (f(x)) where L is a loss
minimizing a loss function
being dissimilar from x, such as the mean squared error.
for
decoder is linear and L is the mean squared error, an undercomplete
When the Component Analysis (PCA).
same subspace as Principal
autoencoder learns to span the task has learned the principal
trained to perform the copying
In this case, an autoencoder
side-effect.
subspace of the training data as a PCA that
dimensionality reduction, we tend to think of methods like
When we think of
hyperplane to represent data in a higher-dimensional form
form a lower-dimensional
without losing information. disadvantage
only build linear relationships. As a result, it is put at a
" However, PCA can autoencoders that can learn non-linear
undercomplete
compared with methods like
dimensionality reduction.
relationships and therefore, perform better in

knowledge
NICAL P IBUCATONS - an up-thrust for
Deep Leaming Autoencoders and Generative Models
5-7

Inhs tom of nonlinear dimensionality reduction where the autoencoder learns a non
linear manifold is also termed as manifold learning
Effectively, if we remove allnon-linear activations from an undercomplete autoencoder
and use only linear lavers we reduce the undercomplete autoencoder into something
that works at an equal footing with PCA.
Ine loss function used to train an ndercomplete autoencoder is called reconstruction
loss, as it is a check of how well the image has been reconstructed from the input.

5.2 Regularized Autoencoders

the input dimension and
n undercomplete autoencoders the code dimension is less than
distribution. These autoencoders
ey lean to capture most salient features of the data
the encoder and
Tails in learning anything useful about the dataset if the capacity of
decoder is too great.
overcomplete but still learn something
"A regular1zed autoencoder can be nonlinear and
capacity is great enough to learm a
useful about the data distribution even if the model
trivial identity function.
of autoencoder should be able to trained successfully by choosing the
" Any architecture decoder based on the complexity
and the capacity of the encoder and
dimension of code
distribution to be modeled. Regularized autoencoders provides this ability.
of
model capacity by keeping the encoder and decoder shallow and
" Instead of limiting the that encourages the
autoencoders use a loss function
the code size small, regularized its input to its output.
to the ability to copy
model to have other properties in addition
5.2.1 Sparse Autoencoders
nodes greater than input nodes. They can still discove
Sparse autoencoders have hidden autoencoder is visualized where th
generic sparse
important features from the data. A
corresponds with the level of activation. Sparse autoencod
obscurity of a node small amount are allowed to
than inputs, but only a
includes more hidden nodes
active at once.
equal numbers of inpL
5.2.1 shows simple single-layer sparse auto encoder with
" Fig.
nodes (h).
(x), outputs (xhat) and hidden
 Autooicoders and Generative Mocdols
Deep Leaming

Fig. 5.2.1 Simple single-layer sparse auto encoder

Sparsity constraint is introduced on the hidden layer. This is to prevent output layer
copy input data.
Sparsity may be obtained by additional terms in the loss function during the training
process, either by comnparing the probability distribution of the hidden unit activations
with some low desired value, or by manually zeroing all but the strongest hidden unit
activations.
Advantages :
1. Sparse autoencoders have a sparsity penalty, a value close to zero but not exactly
zero. Sparsity penalty is applied on the hidden layer in addition to the reconstruction
error. This prevents overfitting.
2. They take the highest activation values in the hidden layer and zero out the rest of the
hidden nodes.

Disadvantages :
1. For it to be working, it's essential that the individual nodes of a trained model which
activate are data dependent and that different inputs will result in activations of
different nodes through the network.
2 Denoising Autoencoders
Denoising Autoencoders (DAE) are a stochastic version of standard autoencoders that
reduces the risk of learning the identity function. Denoising autoencoders attempt to get
around this risk of identity-function affiliation by introducing noise, i.e. randomly
corrupting input so that the autoencoder must then denoise'" or reconstruct the original
nput.
 peepLoarning
AutoenHfers ang Ganostiye Mist
Denoising autoencoder is atype of autoencoder, which is atype of neural netey sed
for
unsupervised lenrning.
input before providing it toDenoising refers to intentionally adding nosse te) the raw
the network. Denoising can he achieved using stochastic
mapping.
" Keeping the code
layer snall forced our autoencoder to learn an intell1gent
representation of the data. There is another way to force the autoencoder to learn useful
features, which is adding
noise-free random noise to its inputs and making it recover the original
data. This way the autoencoder can't simply copy the input to its output
because the input also contains
proauce the underlying meaningfulrandom noise. We are asking itto subtract the noise and
data. This is called a denoising autoencoder.
Denoising autoencoders are a stochastic version of standard
the risk of learning the autoencoders that reduces
identity function.
n general, the more hidden Javers in an
reduction autoencoder., the more refined this
dimenssonal
Cali oe. However, if an autoencoder has more hidden layers than inputs there
IS a isk the algorithm only learns the identity function during
the output simply equals the input and then training, the point where
" Fig. 5.2.2 shows becomes useless.
denoising autoencoder.

Add noise to the

input image Feed Measure
corrupted reconstruction
input into loss against
autoencoder original image

coder
Decoder
Fig. 5.2.2 Denoising
autoencoder
Denoising autoencoders attempt to get around this risk of
introducing noise, i.e. randomly corrupting input so thatidentity-function affiliation by
"denoise' or reconstruct the original input. the autoencoder must then
" The denoising
where the noiseautoencoder
gets rid of noise by
can be filtered out learning a representation of the input
easily.
 8-10 Autoenooders and Generative Moxlels
difficult, the auteoencoder eef..
While reoving l0ise directly fhom the image seems
manifold (ike i.
this hy napng the input data into a lower-dmensional
undenonplete autocneoders), where filtering of noise becomesmuch easier.
Essenthally. denoising autoencoders work with the help of non-linear dimensionalty
retetion. The loss function generally used in these types of networks isL, or L., loss,
" Denoising autoencoder helps :
1. The hidden layers of the autoencoder learn more robust filters
2. Reduce the risk of overfitting in the autoencoder
3. Prevent the autoencoder from learming asimple identify function.
" The DAE is trained using a supervised learming algorithm and can be used foravariety
of tasks, such as image and spech denoising, anomaly detection and data compression.
5.3 Stochastic Encoders and Decoders
Autoencoders are feedforward networks and use the same loss functions and output unit
that are used in traditional feedforward networks. For designing the output units and the
loss function of afeedforward network, an output distribution p (y |x) is defined and the
negative log-likelihood - log p(y | x) is minimized where yis avector of targets, e.g.
class labels.

But in an autoencoder, target as well as the input is x and still the same strategy can be
applied. So by using same strategy as in feedforward network, we can assume that for a
given code h, decoder is providing aconditional distribution p,ece|h).
Autoencoder can then be trained by minimizing - log Pecoe(X|h) where the exact form
of the loss function depends on the form of Pecoder'
Similar to traditional feedforward networks, linear output units are used to parameterize
the mean of a Gaussian distribution for real valued x. The negative log-likelihood yields
a mean squared error criterion in this case. Binary x values correspond to a Bernoulli
distribution whose parameters are given by a sigmoid output unit, discrete x values
correspond to a softmax distribution and so on.
Given h, the output variables are treatêd as
conditionaly independent so that evaluation of
Pencoder(hx) \Pdecoderlxjh)
probability distribution is inexpensive. For modeling
outputs with correlations, mixture density outputs can
be used.

Fig. 5.3.1 shows the structure of a stochastic Fig. 5.3.1 Structure of a

autoencoder. Both the encoder andthe decoder are not stochastic autoencoder

TECHNICAL PUBLICATIONS -an up-thrust for knowiedge

Deep Learming 5- 11
AutoenCoders and Gerorntivo Modols

Simpie runctions but instead involve some noise iniection, meaning that their output can
be seen as sampled from a distribution.
stochastic encoder
Any latent variable model Pmode(h, x) defines
Peneode (h x) * Pmodehx) and astochastic decoder paecoder(X |h) Pmodex|h).
5.4 Deep Generative Models
new
Generative models learn the distribution of the training data and help in generating
those distribution. In most cases,
tata PointS rom the learned distribution by sampling
Gaussian distribution of the data are assumed.
learning technique that learns the
diet, à deep generative model is an unsupervised
model network.
distribuion of the training data while optinmizing the loss function of the
most of them are used to
1nere are several variants ofthe deen generative models and
are also classified
pertorm dual function viz. abstraction and generation. These models
implicit probability
depending on whether the network is learning an explicit or an
distribution.
rmany hidden layerstrained
Deep Generative Models (DGM) are neural networks with
distributions using a large
to approximate complicated, high-dimensional probability
use the DGMs to estimate the
number of samples. When trained successfulý, we can
the underlying
likelihood of each observation and to create new samples from
distribution.
AutoEncoder (VAE) and Generative
Deep generative models are Variational
Adversarial Nets (GANS).
a decoder. VAÊs can perform
VAEsare generative models consistingg of an encoder and
Gaussian distribution for the data. Most
both abstraction and generation and assume
data samples from the learnt
commonly, a trained VAE can be used for generating new
distribution space.
and a discriminator. GANS are
GANs are generative models consisting of a generator
the generator and the
trained using an adversarial learning technique, where
discriminator compete with each other.
tries to generate more
During this process of training, the generator improves itself and
real dataset.
realistic samples which the discriminator cannot distinguish with the
5.4.1 Boltzmann Machine
nodes make binary
" Boltzmann Machine is a kind of recurrent neural network where the
decisions and are present with certain biases. Several Boltzmann machines can be

TECHNICAL PUBLICA TIONS-an up-thrust for knowledge

 B12
Deep Lnaning
sophisticted systetms ssch as a deep beisef
even more
colaborated fogether to make
that helps them to discover
network
of a fearning algorithm
consist
" Boltzmann mach1ncs composed of binary
vectors.
datasets
interesting features in Boltzimann machine
shows architecture of
" Fg. 5.4 1

Hidden
nodes

Visible
nodes

of Boltzmann machine
Fig.5.4.1 Architecture

machine is a generative unsupervised model, which involves learning a

" Boltzmann
and using it to make inferences about
probability distribution from an original dataset
never before seen data.
to as the visible layer) and one or
Boltzmann machine has an input layer (also referred
the hidden layer).
several hidden layers (also referred to as
" Boltzmann machíne consísts of a neural
network with an input layer and one or severa!
make stochastic decisions about
hidden lavers. The neurons in the neural network
training and the cost function
whether toturm on or off based on the data we feed during
the Boltzmann machine is trying to minimize.
interesting features about the data.
" Bv dong so, the Boltzmann Machine discovers
which help model the complex underlying relationships and patterns present in the data.
feature
" The learning algorithm is generally slow in networks with many layers of
detectors but can be made faster by implementing a learning layer of feature detectors.
" Boltzmann machines are typically used to solve different computational problems such
as. for a search problem, the weights present on the connections can be fixed and are
used to represent the cost function of the optimization problem.
Similarly, for alearning problem, the Boltzmann machine can be presented with aset of
binary data vectors from which it must find the weights on the connections so that the
data vectors are good solutions to the optimization problem defined by those weights.

TECHNICAL PUBLICATIONS. an up-thrust for knowedge

 poep Loarnng Autoencoders and Generathve Molais
wt
5. 13
Boltzmann machines make many small updates to their weights and each update
requires solving many different search problems.
The main purpose of the Boltzmann Machíne is tooptimize the solution of a problem. It
optimizes the weights and quantities related to the particular problem assigned to it. This
method is used when the main objective is to create mapping and learn from the
attributes and target variables in the data.
When the objective is to jdentify an underlving structure or the pattern witin tne data,
unsupervised leaming methods for this model are considered to be more useu
" Key characteristics of the Boltzmann machine are as follows .
a) There are no connections between the
layers.
b) They employ symmetric and recurring structures.
C) t 15 an algorithm for unsupervised learning, meaning that it draws conclusions rom
the input data without labeled replies.
high
a) In their learning process, RBMs attempt to link low energy states with
probability ones and vice versa.
" The various kinds of Boltzmann machines are :
a) Deep Belief Networks (DBNs)
b) Restricted Boltzmann Machines (RBMS)
c) Deep Boltzmann Machines (DBMs)
5.4.2 Deep Belief Network
generative model that use a deep
" A Deep Belief Network (DBN) is a powerful
(RBM).
architecture of multiple stacks of Restricted Boltzmann Machines
consisting of two
Deep Belief Networks (DBN) is an unsupervised learning algorithm Boltzmann
and Restricted
different types of neural networks - Belief Networks
DBN is also
Machines. In contrast to perceptron and backpropagation neural networks,
amulti-layer belief network.
like a vanilla neural
" Each RBM model performs a non-linear transformation (much
that will serve as
network works) on its input vectors and produces as outputs vectors
DBNs and
input for the next RBM model in the sequence. This allows a lot flexibility to
makes them easier to expand.
Being a generative model allows DBNs to be used in either an unsupervised or a
supervised setting. Meaning, DBNS have the ability to do feature learning extraction and
classification that are used in many applications.
 Deep L.eaning 5. 14 Autoencoders nnd Generative Mocdels

networks have multiple layers of latent variables

Generative models called deep belicf
binary or real, whereas the latent variables are normally
The apparent units may be
binary. between the layers. Although it is possible to create more
There are no connections every unit in cach layer is connectedto every unit in
DBNS, typically
sparsely connected
every neighboring layer.
architecture of DBN.
" Fig. S.4.2 shows Class
labels

Top layer
LR
classifier

-DBN

RBM
layers
Bottom
layer

Input
vectors

Fig. 5.4.2 Architecture of DBN

" DBN can be used to solve unsupervised learning tasks to

reduce the dimensionality of
classification
features. and can also be used to solve supervised learning tasks to build
models or regression model.
In the DBN, we have a hierarchy of layers. The top two layers are the associative
memory, and the bottom layer is the visible units. The arrows pointing towards the laver
closest to thedata point to relationships between all lower layers.
Directed acyclic connections in the lower layers translate associative memory to
observable variables.
The lowest layer of visible units receives input data as binary or actual data. Like RBM,
there are no intra-layer connections in DBN. The hidden units represent features that
encapsulate the data's correlations.
" Amatrix of proportional weights Wconnects two layers. We will link every unit in each
layer to every other unit in the layer above it.

TECHNICAL PUBLICATIONS- an up-thrust for knowiedge

Deep Leaming 5-15 Autoencocders and Genorative MOdots

Working of deep belief network:

" The deep belief network's operational pipeline is as follows :
First, we run numerous steps of Gibbs sampling in the top two hidden layers. The top
two hidden layers define the RBM. Thus, this stage effectively extracts a sample from it.
Then generate a sample from the visible units using a single pass of ancestral sampling
through the rest of the model.
Finally, we will use a single bottom-up pass to infer the values of the latent variables in
each layer. In the bottom layer, greedy pretraining begins with an observed data vector.
It then oppositely fine-tunes thè generative weights.
Greedy learning algorithms are used to train deep belief networks. In the greedy
approach, the algorithm adds units in top-down layers and learns generative weights that
minimize the error on training examples. Gibbs sampling is used to understand
the top
two hidden layers.
5.5 Generative Adversarial Networks
generative modeling used
Generative Adversarial Network is defined as the technique of
training data sets. The newly generated data set
to generate new data sets based on
appears similar to the training data sets.
two parts:
A Generative Adversarial Network (GAN) has
generate plausible data. The generated instances become
a) The generator learns to
discriminator.
negative training examples for the real data. The
discriminator learns to distinguish the generator's fake data from
b) The producing implausible results.
discriminator penalizes the generator for
architecture of GAN.
Fig. 5.5.1shows basic Update
Real image samples

Real Discriminator

Real/ Fake

Fake

Update
architecture of GAN
Fig. 5.5.1 Basic
knowledge
an up-thrust for
TECHNICAL PUBLICATIONSS
 Deep Learning 7-16

" The veneratoy ereates an image from arandom seed Ihe discriminator evalates he
imagc hased on its training to see if it can telH reai trom fake. The result gres hack to fhe
generator anddiscriminator so that they improve.
The genertor model takes a fixed-length random vector as input and
in the domain.
generates a 14nole
.The vector is dravwn from randomly from aGaussian distribution, and the
vector is sed
to seed the generative process. After training, points in this
multidimensional vector
space will correspond to points in the problem domain. forming a cornpressed
representation of the data distribution.
" This vector space is refered to as a latent space, ot a vector space comprised of latent
variables. Latent variables or hidden variables are those variables that are important for
adomain but are not directly observable.

5.5.1 Types of GAN

" Different types of Generative Adversarial Networks (GANS) are :
a) Vanilla GAN
b) Conditional GAN (CGAN)
c) Deep Convolutional GAN (DCGAN)
d) CycleGAN
e) Style GAN.
1. Vanilla GAN
" The Vanilla GAN is the most basic sort of GAN, consisting only of the generator and
discriminator. Internally, the generator and discriminator use multi- layer perceptions to
generate and classify images.
The discriminator seeks to determine the likelihood that the input belongs to a particular
class while the generator captures the distribution of the data.
After computing the loss function, feedback is then sent to both the generator and
discriminator and this is where the effort to minimize the loss is made.
2. Conditional GAN
" In this GAN, extra information which might be a class label or
any modal data is given
tothe generator and discriminator.
" The additional information, as the name implies, aids the
discriminator in determining
the conditional probability as opposed to the joint
probability.

TECHNICAL PUBLICATIONS- an up-thrust for knowledge

 5- 17 Autoencoders and Generative Modela
Deep Leeming k

3. Deep ConvolutionalGAN (DCGAN)

was employed.
This is the first GAN where the deep convolutional network generator
producing high resolution and high quality images for ditterentiation.
Leaky-ReLu activation function, while in the
" In the Discriminator, all lavers utilize the
exception of the final layer, which
Generator, all layers use ReLU activation with the
uses Tanh activation.

4. CycleGAN
designed for mapping one image to another image or image-to-image
" Ihis GAN is
translations.
and winter are subjected to the Image-Image translation process,
" For instance, if summer
mapping function that could transform summer images into those of
we discOver a features in accordance with the
versa by adding or removing
WInter mages and vice
such that the predicted output and actual output have the least amount
mapping function,
of loss.

5.6 Variational Autoencoders

is
autoencoder can be defined as being an autoencoder whose training
Variational
overfitting and ensure that the latent space has good properties that
regularised to avoid
enable generative process.
architecture composed of both an encoder and a decoder
Variational autoencoder is an encoded-decoded
minimize the reconstruction error between the
and that is trained to
data and the initial data.
provides a probabilistic manner for describing an
(VAE)
A Variational Autoencoder building an encoder which outputs a single
than
observation in latent space. Thus,rather will formulate our encoder to describe a
state attribute, we
value to describe each latent
attribute.
probability distribution for each latent unsupervised
generative deep learning model that is capable of
" VAE is a kind of models to unlabeled data. A
is the process of fitting
learning. Unsupervised learning unsupervised learning model that is capable of
of
generative model is a specific kind
not seen in training.
generating new data points that were
 AuloeCOders And
A-8 Generotive Modols
57 Two Marks Questions with Answers
reconstruct
What is n atoencoder
that canleam
to compress and
of input data, such
temorks model consists two
Ans:Attis ane neuralI An
autoencodet
neuns.
parts : An
laver of

representation
What is atm of autoencoder ?
aMeNOteris to leam a
lower-dimensional
by training the
network
to
(encoding for a
dimensionality reduction, capture
Nihesinal data. ypically Nr
image.
te mst imgrtant arts ofthe input
autoencoder ? encourages the model
to
What is regularization in function that have other
a loss
Ans. Regularized autoenNOoders use
output.
Rwnes sides opving its input to its
autoencoder supervised or unsupervised ? compressed
Is
network model that seeksto learn a representation
trained
of
Ans. : An autoenoder is a
neural although technically, they are
an unsupervised learning
method, using
te mut. They are self-supervised.
as
Senisilcaming methods, referred to
autoencoder ?
Q.5 Why do we use representation (encoding) for higher-
lower-dinmensional
to learn a
Ans. : An autoencoder ainns trainingthe network to capture the most
ata, ypically for dimensionalityreduction, by
umensonal
yortant parts of the input image.
network used for ?
Q.6 What is a deep belief associated with
Networks (DBNs) have been used to address the problems
Ans. : Deep Belief to noer
such as slow learning, becoming stuck in local minima owing
ciasssc neural networks,
training datasets.
peraneter selection and requiring many
supervised or unsupervised ?
Q.7 is the deep belief network
unsupervised learning algorithm consisting of two
Ans. : Deep Belief Networks (DBN) is an
and Restricted Boltzmann Machines. In
diterent types of neural nerworks - Belief Networks belief
percepron and backpropagation neural networks, DBN is also a multi-layer
ontrast to

Explain key characteristics of the Boltzmann machine.

Ans. : Key characteristics of the Boltzmann machine are as follows:
There are no connections between the layers.
They employ symmetric and recurring structures.

TECHNICAL PUBLICATIONS- an up-thrust for knowiedge

 mspevised lening,
AutootClerg are serierative Motalfs
neanng that it draws coIclsions fron the
What ia Botamann mpst
Ans.
machine ?
tnttzman maehune is a kind of
reeurrent neural
AINNs and aIr present with
certain biases, Several network where the nodes make binary
yrthe mae evn more Boltzmann
lnamg algnhun that helps sophisticated Nyste)s sueh as a deep machines
belief
can be
collaborated
them to discover network. It consist of a
interesting features in datasets composed of
a.i0 Define binary
Ans. generative adversarlal networks.
Generative adversarial network is defined as
senerate new data setsbased on training data the technique of
the traning generative