UNIT - V

Optimization for Train Deep Models: Challenges in Neural Network Optimization, Basic
Algorithms, Parameter Initialization Strategies, Algorithms with Adaptive Learning Rates,
Approximate Second Order Methods, Optimization Strategies and Meta-Algorithms

Applications: Large-Scale Deep Learning, Computer Vision, Speech Recognition, Natural

Language Processing

The Challenges of Optimizing Deep Learning Models

There are several types of optimization in deep learning algorithms but the most interesting ones
are focused on reducing the value of cost functions.

Some Basics of Optimization in Deep Learning Models

The core of deep learning optimization relies on trying to minimize the cost function of a
model without affecting its training performance. That type of optimization problem contrasts
with the general optimization problem in which the objective is to simply minimize a specific
indicator without being constrained by the performance of other elements(
ex:training).

Most optimization algorithms in deep learning are based on gradient estimations. In that
context, optimization algorithms try to reduce the gradient of specific cost functions evaluated
against the training dataset. There are different categories of optimization algorithms
depending on the way they interact with the training dataset. For instance, algorithms that use
the entire training set at once are called deterministic. Other techniques that use one training
example at a time has come to be known as online algorithms. Similarly, algorithms that use
more than one but less than the entire training dataset during the optimization process are
known as minibatch stochastic or simply stochastic.
The most famous method of stochastic optimization which is also the most common algorithm
in deep learning solution is known as stochastic gradient descent(SGD)(read my previous
article about SGD).

Regardless of the type of optimization algorithm used, the process of optimizing a deep learning
model is a careful path full of challenges.

Common Challenges in Deep Learning Optimization

There are plenty of challenges in deep learning optimization but most of them are related to
the nature of the gradient of the model. Below, I’ve listed some of the most common
challenges in deep learning optimization that you are likely to run into:

a)Local Minima: local minima is a permanent challenge in the optimization of any deep
learning algorithm. The local minima problem arises when the gradient encounters many local
minimums that are different and not correlated to a global minimum for the cost function.

B.saddle points

saddle points are another reason for gradients to vanish. A saddle point is any location
where all gradients of a function vanish but which is neither a global nor a local minimum.

Flat Regions: In deep learning optimization models, flat regions are common areas that
represent both a local minimum for a sub-region and a local maximum for another. That
duality often causes the gradient to get stuck.
c) Inexact Gradients: There are many deep learning models in which the cost function is
intractable which forces an inexact estimation of the gradient. In these cases, the inexact gradients
introduce a second layer of uncertainty in the model.

d) Local vs. Global Structures: Another very common challenge in the optimization of deep
leavening models is that local regions of the cost function don’t correspond with its global
structure producing a misleading gradient.
Vanishing and Exploding
Gradients
Deep learning networks can be problematic when the numbers change too quickly or slowly
through many layers. This can make it hard for the network to learn and stay stable. This can
cause difficulties for the network in learning andble.
remaining sta

Solution
: Gradient clipping, advanced weight initialization, and skip connections help a
computer learn things accurately and consistently.

Overfitting
Overfitting happens when a model knows too much about the training data, so it can't make
good predictions about new data. As a result, the model performs well on the training data
but struggles to make accurate predictions on new, unseen data. It's essential to address
overfitting by employing techniques like regularization,
-validation,
and
crossmore diverse
datasets to ensure the model generalizes well to unseen examples.
Regularisation techniques help us ensure our models memorize the data and use what
they've learned to make good predictions about new data. Techniques like dropout, L1/L2
regularisation, and early stopping can help us do this.

Data Augmentation and Preprocessing

Data augmentation and preprocessing are techniques used to provide better information to the
model during training, enabling it to learn more effectively and make accurate predictions.
Solution: Apply data augmentation techniques like rotation, translation, and flipping
alongside data normalization and proper handling of missing values.
Label Noise
Training data sometimes need to be corrected, making it hard for computers to do things well.
Solution: Using special kinds of math called "loss functions" can help ensure that the model
you are using is not affected by label mistakes.
Imbalanced Datasets
Datasets can have too many of one type of thing and need more of another type. This can
cause models not to work very well for things not represented as much.
Solution: Classes can sometimes be uneven, meaning more people are in one group than
another. To fix this, we can use special techniques like class weighting, oversampling, or data
synthesis to ensure that all the classes have the same number of people.
Computational Resource Constraints
Training deep neural networks can be very difficult and take a lot of computer power,
especially if the model is very big.
Solution: Using multiple computers or special chips called GPUs and TPUs can help make
learning faster and easier.
Hyperparameter Tuning
Deep neural networks have numerous hyperparameters that require careful tuning to achieve
optimal performance.
Solution: To efficiently find the best hyperparameters, such as Bayesian optimization or
genetic algorithms, utilize automated hyperparameter optimization methods. Convergence
Speed
It is important to ensure a model works quickly when using lots of data and complicated
designs.
Solution: Adopt learning rate scheduling or adaptive algorithms like Adam or RMSprop
to expedite convergence. Memory Constraints
Computers need a lot of memory to train large models and datasets, but they can work
properly if there is enough memory.
Solution: Reduce memory usage by applying model quantization, using mixed-precision
training, or employing memory-efficient architectures like MobileNet or EfficientNet.
Transfer Learning and Domain Adaptation
Deep learning networks need lots of data to work well. If they don't get enough data or the
data is different, they won't work as well.
Solution: Leverage transfer learning or domain adaptation techniques to transfer knowledge
from pre-trained models or related domains.
.
Adversarial Attacks
Deep neural networks are unique ways of understanding data. But they can be tricked by
minimal changes that we can't see. This can make them give wrong answers.
Solution: Employ adversarial training, defensive distillation, or certified robustness methods
to enhance the model's robustness against adversarial attacks.
Interpretability and Explainability
Understanding the decisions made by deep neural networks is crucial in critical applications
like healthcare and autonomous driving.
Solution: Adopt techniques such as LIME (Local Interpretable Model-Agnostic
Explanations) or SHAP (SHapley Additive exPlanations) to explain model predictions.
Handling Sequential Data
Training deep neural networks on sequential data, such as time series or natural language
sequences, presents unique challenges.
Solution: Utilize specialized architectures like recurrent neural networks (RNNs) or
transformers to handle sequential data effectively.
Limited Data
Training deep neural networks with limited labeled data is a common challenge, especially in
specialized domains.
Solution: Consider semi-supervised, transfer, or active learning to make the most of available
data.

Catastrophic Forgetting
When a model forgets previously learned knowledge after training on new data, it encounters
the issue of catastrophic forgetting.
Solution: Implement techniques like elastic weight consolidation (EWC) or knowledge
distillation to retain old knowledge during continual learning.
Hardware and Deployment Constraints
Using trained models on devices with not much computing power can be hard.
Solution: Scientists use special techniques to make computer models run better on devices
with limited resources. Data Privacy and Security
When training computers to do complex tasks, it is essential to keep data private and ensure
the computers are secure.
Solution: Employ federated learning, secure aggregation, or differential privacy techniques to
protect data and model privacy.
Long Training Times
Training deep neural networks is like doing a challenging puzzle. It takes a lot of time to
assemble the puzzle, especially if it is vast and has a lot of pieces.
Solution: Special tools like GPUs or TPUs can help us train our computers faster. We can
also try using different computers simultaneously to make the training even quicker.
Exploding Memory Usage
Some models are too big and need a lot of space, so they are hard to use on regular
computers.
Solution: Explore memory-efficient architectures, use gradient checkpointing, or consider
model parallelism for training.
Learning Rate Scheduling
Setting an appropriate learning rate schedule can be challenging, affecting model convergence
and performance.
Solution: Using special learning rate schedules can help make learning easier and faster.
These schedules can be used to help teach things in a better way.
Avoiding Local Minima
Deep neural networks can get stuck in local minima during training, impacting the model's
final performance.
Solution: Using unique strategies like simulated annealing, momentum-based optimization,
and evolutionary algorithms can help us escape difficult spots.
Unstable Loss Surfaces
Finding the best way to do something can be very hard when there are many different options
because the surface it is on is complicated and bumpy.
Solution: Utilize weight noise injection, curvature-based optimization, or geometric methods
to stabilize loss surfaces.
Ill-Conditioned Matrix

In neural network the adjustments of weights computation and calculation in hidden layer
when calculate in matrix form it simply tells us the characteristics of the matrix in terms of
further computations and calculations, or formally it can be defined as a measure of how
much the output value of the function can change for a small change in the input argument.

A matrix is said to be Ill-conditioned if the condition number is very high, so for a small
change in the input function/the Hessian matrix (The Hessian Matrix is a square matrix of
second ordered partial derivatives of a scalar function. It is of immense use in linear algebra
as well as for determining points of local maxima or minima. ) we will end up getting
outputs with high variance

Basic Algorithms

Gradient Descent is an iterative optimization process that searches for an objective function’s
optimum value (Minimum/Maximum). It is one of the most used methods for changing a
model’s parameters in order to reduce a cost function in machine learning projects.

The primary goal of gradient descent is to identify the model parameters that provide the
maximum accuracy on both training and test datasets

Stochastic Gradient Descent (SGD) is a variant of the Gradient

Descent algorithm that is used for optimizing machine learning models. It addresses the
computational inefficiency of traditional Gradient Descent methods when dealing with large
datasets in machine learning projects.

In SGD, instead of using the entire dataset for each iteration, only a single random training
example (or a small batch) is selected to calculate the gradient and update the model
parameters. This random selection introduces randomness into the optimization process,
hence the term “stochastic” in stochastic Gradient Descent
The advantage of using SGD is its computational efficiency, especially when dealing with
large datasets. By using a single example or a small batch, the computational cost per iteration
is significantly reduced compared to traditional Gradient Descent methods that require
processing the entire dataset.

Stochastic Gradient Descent Algorithm

• Initialization: Randomly initialize the parameters of the model.
• Set Parameters: Determine the number of iterations and the learning rate (alpha) for
updating the parameters.
• Stochastic Gradient Descent Loop: Repeat the following steps until the model converges
or reaches the maximum number of iterations:
a. Shuffle the training dataset to introduce randomness.

b. Iterate over each training example (or a small batch) in the shuffled order.

c. Compute the gradient of the cost function with respect to the model parameters using the
current training example (or batch).

d. Update the model parameters by taking a step in the direction of the negative gradient,
scaled by the learning rate.

e. Evaluate the convergence criteria, such as the difference in the cost function between
iterations of the gradient.

• Return Optimized Parameters: Once the convergence criteria are met or the maximum
number of iterations is reached, return the optimized model parameters.

Stochastic gradient descent (SGD) with momentum

the momentum algorithm introduces a variable v that plays the role of velocity—it is the
direction and speed at which the parameters move through parameter space. The velocity is set
to an exponentially decaying average of the negative gradient. The name momentum derives
from a physical analogy, in which the negative gradient is a force moving a particle through
parameter space, according to Newton’s laws of motion. Momentum in physics is mass times
velocity. In the momentum learning algorithm, we assume unit mass, so the velocity vector v
may also be regarded as the momentum of the particle The algorithm is balanced with
momentum and steps velocity is added as

SGD is generally noisier than typical Gradient Descent, it usually took a higher number of
iterations to reach the minima, because of the randomness in its descent. Even though it
requires a higher number of iterations to reach the minima than typical Gradient Descent, it is
still computationally much less expensive than typical Gradient Descent.

Parameter Initialization Strategies

Training algorithms for deep learning models are iterative in nature and require the
specification of an initial point. This is extremely crucial as it often decides whether or not the
algorithm converges and if it does, then does the algorithm converge to a point with high cost
or low cost.

We have limited understanding of neural network optimization but the one property that we
know with complete certainty is that the initialization should break symmetry. This means
that if two hidden units are connected to the same input units, then these should have different
initialization or else the gradient would update both the units in the same way and we don’t
learn anything new by using an additional unit. The idea of having each unit learn something
different motivates random initialization of weights which is also computationally cheaper.

Biases are often chosen heuristically (zero mostly) and only the weights are randomly
initialized, almost always from a Gaussian or uniform distribution. The scale of the
distribution is of utmost concern. Large weights might have better symmetry-breaking effect
but might lead to chaos (extreme sensitivity to small perturbations in the input) and exploding
values during forward & back propagation. As an example of how large weights might lead to
chaos, consider that there’s a slight noise adding ϵ to the input. Now,
we if did just a simple linear transformation like W * x , the ϵ noise

would add a factor of W * ϵ to the output. In case the weights are

high, this ends up making a significant contribution to the output. SGD and its variants tend
to halt in areas near the initial values, thereby expressing a prior that the path to the final
parameters from the initial values is discoverable by steepest descent algorithms.

Various suggestions have been made for appropriate initialization of the parameters. The most
commonly used ones include sampling the
weights of each fully-connected layer having m inputs and n outputs

uniformly from the following distributions:

• U(-1 / √m, 1 / √m)

• U(- √6 / (m+n), √6 / (m+n))

U(a, b) represents the uniform distribution where the probability of each value between a and
b, a and b inclusive, is 1/(b-a). The probability of every other value is 0.
These initializations have already been incorporated into the most commonly used Deep
Learning frameworks nowadays so that you can just specify which initializer to use and the
framework takes care of sampling appropriately. For e.g. Keras, which is a very famous deep
learning framework, has a module called initializers, where the second distribution (among
the 2 mentioned above) is implemented as glorot_uniform .

One drawback of using 1 / √m as the standard deviation is that the weights end up being small
when a layer has too many input/output units. Motivated by the idea to have the total amount
of input to each unit independent of the number of input units m, Sparse initialization sets
each unit to have exactly k non-zero weights. However, it takes a long time for GD to correct
incorrect large values and hence, this initialization might cause problems.

If the weights are too small, the range of activations across the minibatch will shrink as the
activations propagate forward through the network.By repeatedly identifying the first layer
with unacceptably small activations and increasing its weights, it is possible to eventually
obtain a network with reasonable initial activations throughout.

The biases are relatively easier to choose. Setting the biases to zero is compatible with most
weight initialization schemes except for a few cases .

Algorithms with Adaptive Learning Rates

• AdaGrad: it is important to incrementally decrease the learning rate for faster

convergence. Instead of manually reducing the learning rate after each (or several)
epochs, a better approach is to adapt the learning rate as the training progresses. This
can be done by scaling the learning rates
of each model parameter individually inversely proportional to the square root of the
sum of historical squared values of the gradient. In the parameter update equation
below, r is initialized with 0 and the multiplication in the update step happens
element-wise as mentioned. Since the gradient value would be different for each
parameter, the learning rate is scaled differently for each parameter too.
 • Thus, those parameters having a large gradient have a large decrease in the learning
rate as the learning rate might be too high leading to oscillations or it might be
approaching the minima but having large learning rate might cause it to jump over the
minima as explained in the figure below, because of which the learning rate should be
decreased for better convergence, while those with small gradients have a small
decrease in the learning rate as they might have already approached their respective
minima and should not be pushed away from that. Even if they have not, reducing the
learning rate too much would reduce the gradient even further leading to slower
learning.

AdaGrad parameter update equation.

This figure illustrates the need to reduce the learning rate if gradient is large in case of a

single parameter. 1) One step of gradient descent representing a large gradient value. 2)
Result of reducing the learning rate — moves towards the minima 3) Scenario if the learning
rate was not reduced — it would have jumped over the minima.

However, accumulation of squared gradients from the very beginning can lead to excessive
and premature decrease in the learning rate. Consider that we had a model with only 2
parameters (for simplicity) and both the initial gradients are 1000.
After some iterations, the gradient of one of the

Figure explaining the problem with AdaGrad. Accumulated gradients can cause the learning
rate to be reduced far too much in the later stages leading to slower learning.

parameters has reduced to 100 but that of the other parameter is still around 750. However,
because of the accumulation at each update, the accumulated gradient would still have
almost the same value. For e.g. let the accumulated gradients at each step for the
Parameter 1 be 1000 + 900 + 700 + 400 + 100 = 3100,
1/3100=0.0003 and that for Parameter 2 be: 1000 + 900 + 850 + 800 + 750 = 4300, 1/4300 =
0.0002. This would lead to a similar decrease in the learning rates for both the parameters,
even though the parameter having the lower gradient might have its learning rate reduced too
much leading to slower learning.

• RMSProp: RMSProp addresses the problem caused by accumulated gradients in

AdaGrad. It modifies the gradient accumulation step to an exponentially weighted
moving average in order to discard history from the extreme past. The RMSProp
update is given by:

ρ is the weighing used for exponential averaging. As more updates are made, the contribution
of past gradient values are reduced since ρ < 1 and ρ > ρ² >ρ³ …
This allows the algorithm to converge rapidly after finding a convex bowl, as if it were an
instance of AdaGrad initialized within that bowl. . Consider the figure below. The region
represented by 1 indicates usual RMSProp parameter updates as given by the update
equation, which is nothing but exponentially averaged AdaGrad updates. Once the
optimization process lands on A, it essentially lands at the top of a convex bowl. At this
point, intuitively, all the updates before A can be seen to be forgotten due to the exponential
averaging and it can be seen as if (exponentially averaged) AdaGrad updates start from
point A onwards.

Intuition behind RMSProp. 1) Usual parameter updates 2) Once it reaches the convex bowl,
exponentially weighted averaging would cause the effect of earlier gradients to reduce and to
simplify, we can assume their contribution to be zero. This can be seen as if AdaGrad had
been used with the training initiated inside the convex bowl

• Adam: Adapted from “adaptive moments”, it focuses on combining RMSProp and

Momentum. Firstly, it views Momentum as an estimate of the first-order moment and
RMSProp as that of the second moment. The weight update for Adam is given by:

Secondly, since s and r are initialized as zeros, the authors observed a bias during the initial
steps of training thereby adding a correction term for both the moments to account for their
initialization near the origin. As an example of what the effect of this bias correction is, we’ll
look at the values of s and r for a single parameter (in which case everything is now
represented as a scalar). Let’s first understand what would happen if there was no bias
correction. Since s (notice that this is not in bold as we are looking at the value for a single
parameter and the s here is a scalar) is initialized as zero, after the first iteration, the value of
s would be (1 — ρ1) * g and that of r would be (1 — ρ2) * g². The preferred values for ρ1
and ρ2 are 0.9 and 0.99 respectively. Thus, the initial values of s and r are pretty small and
this gets compounded as the training progress. However, if we now use bias correction, after
the first iteration, the value of s is just g and that of r is just g². This gets rid of the bias that
occurs in the initial phase of training. A major advantage of Adam is that it’s fairly robust to
the choice of these hyperparameters, i.e. ρ1 and ρ2.

3. Approximate Second-Order Methods

The optimization algorithms that we’ve looked at till now involved computing only the first
derivative. But there are many methods which involve higher order derivatives as well. The
main problem with these algorithms are that they are not practically feasible in their vanilla
form and so, certain methods are used to approximate the values of the derivatives. We
explain three such methods, all of which use empirical risk as the objective function:

 Newton’s Method: This is the most common higher-order derivative method used. It
makes use of the curvature of the loss function via its second-order derivative to
arrive at the optimal point. Using the second-order Taylor Series expansion to
approximate J(θ) around a point θo and ignoring derivatives of order greater than 2
(this has already been discussed in previous chapters), we get:
We know that we get a critical point for any
f(x) function
by solving
forf'(x) = 0. We get the following critical point of the above
equation (refer toAppendi
the x for proof):

For quadratic surfaces (i.e. where cost function is quadratic), this directly gives the optimal
result in one step whereas gradient descent would still need to iterate. However, for surfaces
that are not quadratic, as long as the Hessian remains positive definite, we can obtain the
optimal point through a 2-step iterative process — 1) Get the inverse of the Hessian and 2)
update the parameters.

Saddle points are problematic for Newton’s method. If all the eigenvalues are not positive,
Newton’s method might cause the updates to move in the wrong direction. A way to avoid
this is to add regularization:

However, if there is a strong negative curvature i.e. the eigenvalues are largely negative, α
needs to be sufficiently high to offset the negative eigenvalues in which case the Hessian
becomes dominated by the diagonal matrix. This leads to an update which becomes the
standard gradient divided by α:

Another problem restricting the use of Newton’s method is the computational cost. It takes
O(k³) time to calculate the inverse of the Hessian where k is the number of parameters. It’s
not uncommon for Deep Neural Networks to have about a million parameters and since the
parameters are updated every iteration, this inverse needs to be calculated at every iteration,
which is not computationally feasible.
  Conjugate Gradients: One weakness of the method of steepest descent (i.e. GD) is
that line searches happen along the direction of the gradient. Suppose the previous
search direction is d(t-1). Once the search terminates (which it does when the
gradient along the current gradient direction vanishes) at the minimum, the next
search direction, d(t) is given by the gradient at that point, which is orthogonal to
d(t1) (because if it’s not orthogonal, it’ll have some component along d(t-1) which
cannot be true as at the minimum, the gradient along d(t-1) has vanished).
Upon getting the minimum along the current search direction, the minimum along
the previous search direction is not preserved, undoing, in a sense, the progress made
in previous search direction.

In the method of conjugate gradients, we seek a search direction that is conjugate to the
previous line search direction:

Now, the previous search direction contributes towards finding the next search direction.

with d(t) and d(t-1) being conjugates if d(t)' H d(t-1) = 0. βt decides how much of d(t-1) is
added back to the current search direction. There are two popular choices for βt — Fletcher-
Reeves and Polak-Ribière. These discussions assumed the cost function to be quadratic where
the conjugate directions ensure that the gradient along the previous direction does not
increase in magnitude. To extend the concept to work for training neural networks, there is
 one additional change. Since it’s no longer quadratic, there’s no guarantee anymore than the
conjugate direction would preserve the minimum in the previous search directions. Thus, the
algorithm includes occasional resets where the method of conjugate gradients is restarted with
line search along the unaltered gradient.

 BFGS: This algorithm tries to bring the advantages of Newton’s method without the
additional computational burden by approximating the inverse of H by M(t), which is
iteratively refined using low-rank updates. Finally, line search is conducted along the
direction M(t)g(t). However, BFGS requires storing the matrix M(t) which takes
O(n²) memory making it infeasible. An approach called Limited Memory BFGS (L-
BFGS) has been proposed to tackle this infeasibility by computing the matrix M(t)
using the same method as BFGS but assuming that M(t−1) is the identity matrix.

4. Optimization Strategies and Meta-Algorithms

• Batch Normalization: Batch normalization (BN) is one of the most exciting

innovations in Deep learning that has significantly stabilized the learning process and
allowed faster convergence rates. The intuition behind batch normalization is as
follows: Most of the Deep Learning networks are compositions of many layers (or
functions) and the gradient with respect to one layer is taken considering the other
layers to be constant. However, in practise all the layers are updated simultaneously
and this can lead to unexpected results. For example, let y* = x W¹ W² … W¹ ⁰. Here,
y* is a linear function of x but not a linear function of the weights. Suppose the
gradient is given by g and we now intend to reduce y* by 0.1. Using first-order Taylor
Series approximation, taking a step
of ϵg would reduce y* by ϵg’ g. Thus, ϵ should be 0.1/(g’

g) just using the first-order information. However, higher order effects also creep in
as the updated y* is given by:

An example of a second-order term would be ϵ² g1 g2 ∏ wi. ∏ wi can be negligibly small or

exponentially high depending on whether the individual weights are less than or greater than
1. Since the updates to one layer is so strongly dependent on the other layers, choosing an
appropriate learning rate is tough. Batch normalization takes care of this problem by using an
efficient reparameterization of almost any deep network. Given a matrix of activations, H, the
normalization is given by: H’ = (H-μ) / σ, where the subtraction and division is broadcasted.

is added to ensure that σ is not equal to 0.

Going back to the earlier example of y*, let the activations of layer l be given by h(l-1). Then
h(l-1) = x W1 W2 … W (l-1). Now, if x is drawn from a unit Gaussian, then h(l-1) also
comes from a Gaussian, however, not of zero mean and unit variance, as it is a linear
transformation of x. BN makes it zero mean and unit variance. Therefore, y* = Wl h(l-1) and
thus, the learning now becomes much simpler as the parameters at the lower layers mostly do
not have any effect. This simplicity was definitely achieved by rendering the lower layers
useless. However, in a realistic deep network with nonlinearities, the lower layers remain
useful. Finally, the complete reparameterization of BN is given by replacing H with γH’ + β.
This is done to retain its expressive power and the fact that the mean is solely determined by
XW. Also, among the choice of normalizating X or XW + B, the authors recommend the
latter, specifically XW, since B becomes redundant because of β. Practically, this means that
when we are using the Batch Normalization layer, the biases should be turned off. In a deep
learning framework like Keras, this can be done by setting the parameter use_bias=False in
the Convolutional layer.

• Coordinate Descent: Generally, a single weight update is made by taking the

gradient with respect to every parameter. However, in cases where some of the
parameters might be independent (discussed below) of the remaining, it might be
more efficient to take the gradient with respect to those independent sets of
 parameters separately for making updates. Let me clarify that with an example.
Suppose we have the following cost function:

This cost function describes the learning problem called sparse coding. Here, H refers to the
sparse representation of X and W is the set of weights used to linearly decode H to retrieve X.
An explanation of why this cost function enforces the learning of a sparse representation of X
follows. The first term of the cost function penalizes values far from 0 (positive or negative
because of the modulus, |H|, operator. This enforces most of the values to be 0, thereby
sparse. The second term is pretty self-explanatory in that it compensates the difference
between X and H being linearly transformed by W, thereby enforcing them to take the same
value. In this way, H is now learned as a sparse “representation” of X. The cost function
generally consists of additionally a regularization term like weight decay, which has been
avoided for simplicity. Here, we can divide the entire list of parameters into two sets, W and
H. Minimizing the cost function with respect to any of these sets of parameters is a convex
problem. Coordinate Descent (CD) refers to minimizing the cost function with respect to
only 1 parameter at a time. It has been shown that repeatedly cycling through all the
parameters, we are guaranteed to arrive at a local minima. If instead of 1 parameter, we take a
set of parameters as we did before with W and H, it is called block coordinate descent (the
interested reader should explore Alternating Minimization). CD makes sense if either the
parameters are clearly separable into independent groups or if optimizing with respect to
certain set of parameters is more efficient than with respect to others.
The points A, B, C and D indicates the locations in the parameter space where coordinate
descent landed after each gradient step.

Coordinate descent may fail terribly when one variable influences the optimal value of
another variable.

• Polyak Averaging: Polyak averaging consists of averaging several points in the

parameter space that the optimization algorithm traverses through. So, if the algorithm
encounters the points θ(1), θ(2), … during optimization, the output of Polyak
averaging is:

The figure below explains the intuition behind Polyak averaging:

The optimization algorithm might oscillate back and forth across a valley without ever
reaching the minima. However, the average of those points should be closer to the bottom of
the valley.
 Most optimization problems in deep learning are non-convex where the path taken by the
optimization algorithm is quite complicated and it might happen that a point visited in the
distant past might be quite far from the current point in the parameter space. Thus, including
such a point in the distant past might not be useful, which is why an exponentially decaying
running average is taken. This scheme where the recent iterates are weighted more than the
past ones is called Polyak-Ruppert Averaging:

• Supervised Pre-training: Sometimes it’s hard to directly train to solve for a specific
task. Instead it might be better to train for solving a simpler task and use that as an
initialization point for training to solve the more challenging task.

Applications: Large-Scale Deep Learning : Computer Vision, Speech Recognition, Natural

Language Processing

Common Applications of Deep Learning

Deep learning has many uses in many fields, and its potential grows. Let’s analyze a few of
artificial intelligence’s widespread profound learning uses.

• Image Recognition and Computer Vision

• Natural Language Processing (NLP)

• Speech Recognition and Voice Assistants

• Recommendation Systems

• Autonomous Vehicles

• Healthcare and Medical Imaging

• Fraud Detection and Cybersecurity

• Gaming and Virtual Reality

Image Recognition and Computer Vision

The performance of image recognition and computer vision tasks has significantly improved
due to deep learning. Computers can now reliably classify and comprehend images owing to
training deep neural networks on enormous datasets, opening up a wide range of applications.

A smartphone app that can instantaneously determine a dog’s breed from a photo and self-
driving cars that employ computer vision algorithms to detect pedestrians, traffic signs, and
other roadblocks for safe navigation are two examples of this in practice.

Deep Learning Models for Image Classification

The process of classifying photos entails giving them labels based on the content of the
images. Convolutional neural networks (CNNs), one type of deep learning model, have
performed exceptionally well in this context. They can categorize objects, situations, or even
specific properties within an image by learning to recognize patterns and features in visual
representations.
Object Detection and Localization using Deep Learning

Object detection and localization go beyond image categorization by identifying and locating
various things inside an image. Deep learning methods have recognized and localized objects
in real-time, such as You Only Look Once (YOLO) and region-based convolutional neural
networks (R-CNNs). This has uses in robotics, autonomous cars, and surveillance systems,
among other areas.
Applications in Facial Recognition and Biometrics

Deep learning has completely changed the field of facial recognition. Hence, allowing for the
precise identification of people using their facial features. Security systems, access control,
monitoring, and law enforcement use facial recognition technology. Deep learning methods
have also been applied in biometrics for functions including voice recognition, iris scanning,
and fingerprint recognition.
Natural Language Processing (NLP)

Natural language processing (NLP) aims to make it possible for computers to comprehend,
translate, and create human language. NLP has substantially advanced primarily to deep
learning, making strides in several language-related activities. Virtual voice assistants like
Apple’s Siri and
Amazon’s Alexa, who can comprehend spoken orders and questions, are a practical
illustration of this.

Deep Learning for Text Classification and Sentiment Analysis

Text classification entails classifying text materials into several groups or divisions. Deep
learning models like recurrent neural networks (RNNs) and long short-term memory (LSTM)
networks have been frequently used for text categorization tasks. To ascertain the sentiment
or opinion expressed in a text, whether good, negative, or neutral, sentiment analysis is a
widespread use of text categorization.

Language Translation and Generation with Deep Learning

Machine translation systems have considerably improved because of deep learning. Deep
learning-based neural machine translation (NMT) models have been shown to perform better
when converting text across multiple languages. These algorithms can gather contextual data
and generate more precise and fluid translations. Deep learning models have also been
applied to creating news stories, poetry, and other types of text, including coherent
paragraphs.

Question Answering and Chatbot Systems Using Deep Learning

Deep learning is used by chatbots and question-answering programs to recognize and reply to
human inquiries. Transformers and attention mechanisms, among other deep learning models,
have made tremendous progress in understanding the context and semantics of questions and
producing pertinent answers. Information retrieval systems, virtual assistants, and customer
service all use this technology.

Speech Recognition and Voice Assistants

The creation of voice assistants that can comprehend and respond to human speech and the
advancement of speech recognition systems have significantly benefited from deep learning.
A real-world example is using your smartphone’s voice recognition feature to dictate
messages rather than typing them and asking a smart speaker to play your favorite tunes or
provide the weather forecast.

Deep Learning Models for Automatic Speech Recognition

Systems for automatic speech recognition (ASR) translate spoken words into written text.
Recurrent neural networks and attention-based models, in particular, have substantially
improved ASR accuracy. Better voice commands, transcription services, and accessibility
tools for those with speech difficulties are the outcome. Some examples are voice search
features in search engines like Google, Bing, etc.

Voice Assistants Powered by Deep Learning Algorithms

Daily, we rely heavily on voice assistants like Siri, Google Assistant, and Amazon Alexa.
Guess what drives them? Deep learning it is. Deep learning techniques are used by these
intelligent devices to recognize and carry out spoken requests. The technology also enables
voice assistants to recognize speech, decipher user intent, and deliver precise and pertinent
responses thanks to deep learning models.

Applications in Transcription and Voice-Controlled Systems

Deep learning-based speech recognition has applications in transcription services, where large
volumes of audio content must be accurately converted into text. Voice-controlled systems,
such as smart homes and incar infotainment systems, utilize deep learning algorithms to
enable handsfree control and interaction through voice commands.

Recommendation Systems
Recommendation systems use deep learning algorithms to offer people personalized
recommendations based on their tastes and behavior.

Deep Learning-Based Collaborative Filtering

A standard method used in recommendation systems to suggest products/services to users based on

how they are similar to other users is collaborative filtering. Collaborative filtering has improved
accuracy and performance thanks to deep learning models like matrix
factorization and deep autoencoders, which have produced more precise and individualized
recommendations.

Personalized Recommendations Using Deep Neural Networks

Deep neural networks have been used to identify intricate links and patterns in user behavior
data, allowing for more precise and individualized suggestions. Deep learning algorithms can
forecast user preferences and make relevant product, movie, or content recommendations by
looking at user interactions, purchase history, and demographic data. An instance of this is
when streaming services recommend films or TV shows based on your interests and history.

Applications in E-Commerce and Content Streaming Platforms

Deep learning algorithms are widely employed to fuel recommendation systems in e-

commerce platforms and video streaming services like Netflix and Spotify. These programs
increase user pleasure and engagement by assisting users in finding new goods,
entertainment, or music that suits their tastes and preferences.
Autonomous Vehicles

Deep learning has significantly impacted how well autonomous vehicles can understand and
navigate their surroundings. These vehicles can analyze enormous volumes of sensor data in
real-time using powerful deep learning algorithms. Thus, enabling them to make wise
decisions, navigate challenging routes, and guarantee the safety of passengers and
pedestrians. This game-changing technology has prepared the path for a time when driverless
vehicles will completely change how we travel.

Deep Learning Algorithms for Object Detection and Tracking

Autonomous vehicles must perform crucial tasks, including object identification and tracking,
to recognize and monitor objects like pedestrians, cars, and traffic signals. Convolutional and
recurrent neural networks (CNNs) and other deep learning algorithms have proved essential
in obtaining high accuracy and real-time performance in object detection and tracking.

Deep Reinforcement Learning for Decision-Making in Self-Driving Cars

Autonomous vehicles are designed to make complex decisions and navigate various traffic
circumstances using deep reinforcement learning. This technology is profoundly used in self-
driving cars manufactured by companies like Tesla. These vehicles can learn from historical
driving data and adjust to changing road conditions using deep neural networks. Selfdriving
cars demonstrate this in practice, which uses cutting-edge sensors and artificial intelligence
algorithms to navigate traffic, identify impediments, and make judgments in real time.
Applications in Autonomous Navigation and Safety Systems

The development of autonomous navigation systems that decipher sensor data, map routes,
and make judgments in real time depends heavily on deep learning techniques. These systems
focus on collision avoidance, generate lane departure warnings, and offer adaptive cruise
control to enhance the general safety and dependability of the vehicles.

Healthcare and Medical Imaging

Deep learning has shown tremendous potential in revolutionizing healthcare and medical
imaging by assisting in diagnosis, disease detection, and patient care. Revolutionizing
diagnostics using AI-powered algorithms that can precisely identify early-stage tumors from
medical imaging is an example of how to do this. This will help with prompt treatment
decisions and improve patient outcomes.
Deep Learning for Medical Image Analysis and Diagnosis

Deep learning algorithms can glean essential insights from the enormous volumes of data that
medical imaging systems produce. Convolutional neural networks (CNNs) and generative
adversarial networks (GANs) are examples of deep learning algorithms. They can be
effectively used for tasks like tumor identification, radiology image processing, and
histopathology interpretation.

Predictive Models for Disease Detection and Prognosis

Deep learning models can analyze electronic health records, patient data, and medical pictures
to create predictive models for disease detection, prognosis, and treatment planning.

Applications in Medical Research and Patient Care

Deep learning can revolutionize medical research by expediting the development of new
drugs, forecasting the results of treatments, and assisting clinical decision-making.
Additionally, deep learning-based systems can also improve medical care by helping with
diagnosis, keeping track of patients’ vital signs, and making unique suggestions for dietary
changes and preventative actions.

Fraud Detection and Cybersecurity

Deep learning has become essential in detecting anomalies, identifying fraud patterns, and
strengthening cybersecurity systems.
Deep Learning Models for Anomaly Detection

These systems shine when finding anomalies or outliers in large datasets. By learning from
typical patterns, deep learning models may recognize unexpected behaviors, network
intrusions, and fraudulent operations. These methods are used in network monitoring,
cybersecurity systems, and financial transactions. JP Morgan Chase, PayPal, and other
businesses are just a few that use these techniques.

Deep Neural Networks in Fraud Prevention and Cybersecurity

In fraud prevention systems, deep neural networks have been used to recognize and stop
fraudulent transactions, credit card fraud, and identity theft. These algorithms examine user
behavior, transaction data, and historical patterns to spot irregularities and notify security
staff. This enables proactive fraud prevention and shields customers and organizations from
financial loss. Organizations like Visa, Mastercard, and PayPal use deep neural networks. It
helps improve their fraud detection systems and guarantees secure customer transactions.

Applications in Financial Transactions and Network Security

Deep learning algorithms are essential for preserving sensitive data, safeguarding financial
transactions, and thwarting online threats. Deep learning-based cybersecurity systems can
proactively identify and reduce potential hazards, protecting vital data and infrastructure by
learning and adapting to changing attack vectors over time.
Gaming and Virtual Reality

Deep learning has significantly improved game AI, character animation, and immersive
surroundings, benefiting the gaming industry and virtual reality experiences. A virtual reality
game, for instance, can adjust and customize its gameplay experience based on the player’s
real-time motions and reactions by using deep learning algorithms.

Deep Learning in Game Development and Character Animation

Deep learning algorithms have produced more intelligent and lifelike video game characters.
Game makers may create realistic animations, enhance character behaviors, and make more
immersive gaming experiences by training deep neural networks on enormous datasets of
motion capture data.

Deep Reinforcement Learning for Game AI and Decision-Making

Deep reinforcement learning has changed game AI by letting agents learn and enhance their
gameplay through contact with the environment. Using deep learning algorithms in game AI
enables understanding optimal strategies, adaptation to various game circumstances, and
challenging and captivating gaming.

Applications in Virtual Reality and Augmented Reality Experiences

Experiences in augmented reality (AR) and virtual reality (VR) have been improved mainly
due to deep learning. Deep neural networks are used by VR and AR systems to correctly track
and identify objects, detect movements and facial expressions, and build real virtual worlds,
enhancing the immersiveness and interactivity of the user experience.

Conclusion

In artificial intelligence, deep learning has become a powerful technology that allows robots
to learn and make wise decisions. Deep learning in AI has many uses, from image
identification and NLP to cybersecurity and healthcare. It has substantially improved the
capabilities of AI systems, resulting in innovations across various fields and the disruption of
entire sectors. Common applications of deep learning in AI Accenture leverages deep learning
within its AI initiatives to enhance data analytics, customer experience, and operational
efficiency.