Page 1

DSC13: Design and Analysis of Algorithm

Unit – 5: Limitations of Algorithm Power

Part-2

5.1. Limitations of Algorithm Power

1. Lower-Bound Arguments,

2. Decision Trees,

3. P, NP and NP-Complete Problems.

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 DSC13: Design and Analysis of Algorithm Page 2
Unit – 5: Limitations of Algorithm Power

5.1. Limitations of Algorithm Power

 Introduction
Some problems cannot be solved by any algorithm. Other problems can be solved
algorithmically but not in polynomial time. And even when a problem can be solved in
polynomial time by some algorithms, there are usually lower bounds on their efficiency.
Lower bounds, are estimates on a minimum amount of work needed to solve a problem. In
general, obtaining a nontrivial lower bound even for a simple-sounding problem is a very
difficult task.
The task here is to establish a limit on the efficiency of any algorithm, known or unknown.
Necessities: a careful description of the operations such algorithms are allowed to perform. If
we fail to define carefully the “rules of the game,” so to speak, our claims may end up in the
large dustbin.

5.1.1. Lower-Bound Arguments

 Introduction
Lower bound means; when we want to ascertain the efficiency of an algorithm with respect to
other algorithms for the same problem, it is desirable to know the algorithm that has best
possible efficiency for solving the same problem. Such best possible efficiency is called
lower bound.
Knowing such a lower bound can tell us how much improvement we can hope to achieve in
our quest for a better algorithm for the problem in question.
 Methods of obtaining lower bounds
o Trivial Lower Bounds
o Information-Theoretic Arguments
o Adversary Arguments
o Problem Reduction

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 DSC13: Design and Analysis of Algorithm Page 3
Unit – 5: Limitations of Algorithm Power

a) Trival lower bounds

The simplest method of obtaining a lower-bound class is based on Counting the
- Number of items in the problem’s input that must be processed and
- Number of output items that need to be produced.
Since any algorithm must at least “read” all the items it needs to process and “write” all its
outputs, such a count yields a trivial lower bound.
 Example1: Generating all permutations of n distinct items
Such algorithm must be in Ω(n!) because the size of the output is n!. And this bound is tight
because good algorithms for generating permutations spend a constant time on each of them
except the initial one
 Example2: Find the lower bound for computing the product of two 𝒏 𝒙 𝒏 matrices.
Trivial lower bound for computing the product of two n x n matrices is Ω(𝒏𝟐 ) because any
such algorithm has to process 2𝒏𝟐 elements in the input matrices and generate 𝒏𝟐 elements
of the product. It is still unknown however, whether this bound is tight.

b) Information theoretic arguments

Trival lower bound approach takes into account the size of a problem’s output. The
information-theoretical approach seeks to establish a lower bound based on the amount of
information it has to produce.
 Example:
Consider, game of “guessing” a number selected by somebody, by asking that person
questions with yes/no answers. The amount of uncertainty that any algorithm solving this
problem has to resolve can be measured by [𝒍𝒐𝒈𝟐 𝒏], the number of bits needed to specify a
particular number among the n possibilities. We can think of an answer to each question as
yielding at most 1 bit of information about the algorithm’s output, i.e., the selected number.
Consequently, any such algorithm will need at least [𝒍𝒐𝒈𝟐 𝒏], such steps before it can
determine its output in the worst case.

c) Adversary arguments
Let us revisit the same game of “guessing” a number used to introduce the idea of an
information-theoretic argument. We can prove that any algorithm that solves this problem
must ask at least [𝒍𝒐𝒈𝟐 𝒏] questions in its worst case by playing the role of a hostile adversary
who wants to make an algorithm ask as many questions as possible. The adversary starts by
considering each of the numbers between 1 and n as being potentially selected. (This is
cheating, of course, as far as the game is concerned, but not as a way to prove our assertion.)
After each question, the adversary gives an answer that leaves him with the largest set of
numbers consistent with this and all the previously given answers. This strategy leaves him
with at least one-half of the numbers he had before his last answer. If an algorithm stops

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 DSC13: Design and Analysis of Algorithm Page 4
Unit – 5: Limitations of Algorithm Power

before the size of the set is reduced to 1, the adversary can exhibit a number that could be a
legitimate input the algorithm failed to identify. It is a simple technical matter now to show
that one needs [𝒍𝒐𝒈𝟐 𝒏] iterations to shrink. An n-element set to a one-element set by halving
and rounding up the size of the remaining set. Hence, at least [𝒍𝒐𝒈𝟐 𝒏] questions need to be
asked by any algorithm in the worst case.

d) Problem reduction
There, we discussed getting an algorithm for problem P by reducing it to another problem Q
solvable with a known algorithm. A similar reduction idea can be used for finding a lower
bound. To show that problem P is at least as hard as another problem Q with a known lower
bound, we need to reduce Q to P (not P to Q!). In other words, we should show that an
arbitrary instance of problem Q can be transformed (in a reasonably efficient fashion) to an
instance of problem P, so any algorithm solving P would solve Q as well. Then a lower
bound for Q will be a lower bound for P. Table 11.1 lists several important problems that are
often used for this purpose.

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 Page 5
DSC13: Design and Analysis of Algorithm
Unit – 5: Limitations of Algorithm Power

5.1.2. Decision trees

 Introduction
Decision tree can be used to study the performance of various algorithms like sorting and
searching.

 Decision tree for finding a minimum of three numbers.

Each internal node of a binary decision tree represents a key comparison indicated in the
node, e.g., k < k’. The node’s left subtree contains the information about subsequent
comparisons made if k < k’, and its right subtree does the same for the case of k > k’. (For the
sake of simplicity, we assume throughout this section that all input items are distinct.) Each
leaf represents a possible outcome of the algorithm’s run on some input of size n. Note that
the number of leaves can be greater than the number of outcomes because, for some
algorithms, the same outcome can be arrived at through a different chain of comparisons.
(This happens to be the case for the decision tree in Figure 11.1.) An important point is that
the number of leaves must be at least as large as the number of possible outcomes. The
algorithm’s work on a particular input of size n can be traced by a path from the root to a leaf
in its decision tree, and the number of comparisons made by the algorithm on such a run is
equal to the length of this path. Hence, the number of comparisons in the worst case is equal
to the height of the algorithm’s decision tree.

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 Page 6
DSC13: Design and Analysis of Algorithm
Unit – 5: Limitations of Algorithm Power

 Decision Trees for Sorting

Most sorting algorithms are comparison based, i.e., they work by comparing elements in a list
to be sorted. By studying properties of decision trees for such algorithms, we can derive
important lower bounds on their time efficiencies.
Consider, as an example, a three-element list a, b, c of orderable items such as real numbers
or strings. For the outcome a < c < b obtained by sorting this list as shown in following
figure, the permutation in question is 1, 3, 2. In general, the number of possible outcomes for
sorting an arbitrary n-element list is equal to n!.

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 DSC13: Design and Analysis of Algorithm Page 7
Unit – 5: Limitations of Algorithm Power

5.1.3. P, NP, and NP-Complete problems

 Introduction
In the study of the computational complexity of problems, the first concern of computer
scientists and professionals is whether a given problem can be solved in polynomial time by
some algorithm.

 Definition
We say that an algorithm solves a problem in polynomial time if its worst-case time
efficiency belongs to O(p(n)) where p(n) is a polynomial of the problem’s input size n. (Note
that since we are using big-oh notation here, problems solvable in, say, logarithmic time are
solvable in polynomial time as well.) Problems that can be solved in polynomial time are
called tractable, and problems that cannot be solved in polynomial time are called intractable.

P and NP Problems
Informally, problems that can be solved in polynomial time are called P. A more formal
definition includes in P only decision problems, which are problems with yes/no answers.

a) P problems: Polynomial problems

Class P is a class of decision problems that can be solved in polynomial time by
(deterministic) algorithms. This class of problems is called polynomial (P).

 Nondeterministic algorithms
A nondeterministic algorithm is a two-stage procedure that takes as its input an instance I of a
decision problem and does the following.
- Nondeterministic (“guessing”) stage: An arbitrary string S is generated that can be
thought of as a candidate solution to the given instance I (but may be complete gibberish
as well).
- Deterministic (“verification”) stage: A deterministic algorithm takes both I and S as its
input and outputs yes if S represents a solution to instance I. (If S is not a solution to
instance I , the algorithm either returns no or is allowed not to halt at all.)
We say that a nondeterministic algorithm solves a decision problem if and only if for every
yes instance of the problem it returns yes on some execution. (In other words, we require a
nondeterministic algorithm to be capable of “guessing” a solution at least once and to be able
to verify its validity. And, of course, we do not want it to ever output a yes answer on an
instance for which the answer should be no.)

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 DSC13: Design and Analysis of Algorithm Page 8
Unit – 5: Limitations of Algorithm Power

b) NP problems: Nondeterministic Polynomial problems

Class NP is the class of decision problems that can be solved by nondeterministic polynomial
algorithms. This class of problems is called nondeterministic polynomial.
Most decision problems are in NP. First of all, this class includes all the problems in P :
P ⊆ NP.
This is true because, if a problem is in P, we can use the deterministic polynomial time
algorithm that solves it in the verification-stage of a nondeterministic algorithm that simply
ignores string S generated in its nondeterministic (“guessing”) stage.
 Examples of NP problem: Hamiltonian circuit problem, the partition problem, decision
versions of the travelling salesman, the knapsack, graph colouring.
 Examples of not NP problem: The halting problem,

Polynomially reducible problem:

A decision problem D1 is said to be polynomially reducible to a decision problem D2, if
there exists a function t that transforms instances of D1 to instances of D2 such that:

1. t maps all yes instances of D1 to yes instances of D2 and all no instances of D1 to no

instances of D2.

2. t is computable by a polynomial time algorithm.

c) NP-complete problems
A decision problem D is said to be NP-complete if:
1. it belongs to class NP
2. every problem in NP is polynomially reducible to D
The fact that closely related decision problems are polynomially reducible to each other is not
very surprising.

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

 Page 2
DSC13: Design and Analysis of Algorithm
Unit – 5: Limitations of Algorithm Power

Challenges of Numerical Algorithms

Numerical analysis is usually described as the branch of computer science concerned with
algorithms for solving mathematical problems. This description needs an important
clarification: the problems in question are problems of
“continuous”mathematics—solving equations and systems of equations, evaluating
such functions as sin x and ln x, computing integrals, and so on—as opposed to problems
of discrete mathematics dealing with such structures as graphs, trees, permutations, and
combinations. Our interest in efficient algorithms for mathematical problems stems from
the fact that these problems arise as models of many real-life phenomena both in the
natural world and in the social sciences. In fact, numerical analysis used to be the main area
of research, study, and application of computer science. With the rapid proliferation of
computers in business and everyday-life applications, which deal primarily with storage
and retrieval of information, the relative importance of numerical analysis has shrunk in the
last 30 years. However, its applications, enhanced by the power of modern computers,
continue to expand in all areas of fundamental research and technology. Thus, wherever
one’s interests lie in the wide world of modern computing, it is important to have at
least some understanding of the special challenges posed by continuous mathematical
problems.

We are not going to discuss the variety of difficulties posed by modeling, the task of
describing a real-life phenomenon in mathematical terms. Assuming that this has already
been done, what principal obstacles to solving a mathematical problem do we face? The
first major obstacle is the fact that most numerical analysis problems cannot be solved
exactly.4 They have to be solved approximately, and this is usually done by replacing an
infinite object by a finite approximation.
For example, the value of ex at a given point x can be computed by approximating its
infinite Taylor’s series about x = 0 by a finite sum of its first terms, called the nth-degree

Taylor polynomial:
ex ≈ 1 + x + x2 / 2! + ... + xn / n!

Prof. Nandan G P, Dept. of BCA, SRS FGC - Chitradurga

Page 1 of 1