Lecture

6
Sorting lower bounds on O(n)-time sorting
 Sorting
• We’ve seen a few O(n log(n))-time algorithms.
• MERGESORT has worst-case running time O(nlog(n))
• QUICKSORT has expected running time O(nlog(n))

Can we do better?

Depends on who
you ask…
 An O(1)-time algorithm for sorting:
StickSort
• Problem: sort these n sticks by length.

• Now they
are sorted
this way.
• Algorithm:
• Drop them on a table.
That may have been unsatisfying
• But StickSort does raise some important questions:
• What is our model of computation?
• Input: array
• Output: sorted array
• Operations allowed: comparisons

-vs-

• Input: sticks
• Output: sorted sticks in vertical order
• Operations allowed: dropping on tables

• What are reasonable models of computation?

Today: two (more) models
• Comparison-based sorting model
• This includes MergeSort, QuickSort, InsertionSort
• We’ll see that any algorithm in this model must take at
least Ω(n log(n)) steps.

• Another model (more reasonable than the stick model…)

• BucketSort and RadixSort
• Both run in time O(n)
Comparison-based sorting
 Comparison-based sorting algorithms
is shorthand for
“the first thing in the input list”

Want to sort these items.

There’s some ordering on them, but we don’t know what it is.

Is bigger than ?

YES There is a genie who knows what

the right order is.

The algorithm’s job is to The genie can answer YES/NO

output a correctly sorted questions of the form:
Algorithm list of all the objects. is [this] bigger than [that]?
 All the sorting algorithms we
have seen work like this.
Pivot!
eg, QuickSort:
7 66 33 5 1 4 2

Is 7 bigger than 5 ? YES

Is 6 bigger than 5 ? YES
Is 3 bigger than 5 ? NO

5 etc.
 Lower bound of Ω(n log(n)).
• Theorem:
• Any deterministic comparison-based sorting algorithm must
take Ω(n log(n)) steps.
• Any randomized comparison-based sorting algorithm must
take Ω(n log(n)) steps in expectation.

• How might we prove this?

1. Consider all comparison-based algorithms, one-by-one,
and analyze them.
Instead, argue that all comparison-based sorting
algorithms give rise to a decision tree.
2. Don’t do that. Then analyze decision trees.
Decision trees

Sort these three things. ≤ ?

YES NO

etc…
≤ ?
YES NO

≤ ?
YES NO
 All comparison-based algorithms look like this
Pivot!
≤ ?
NO Example: Sort these
YES
three things using
QuickSort.
L R L R

etc... ≤ ?
YES NO Pivot!

L L R Now
R
Then we’re done Return
(after some base-
≤ ? recurse
on R
YES NO
case stuff)

L R L R
Return Return
In either case, we’re done
(after some base case stuff and
returning recursive calls).
 All comparison-based algorithms
have an associated decision tree.
The leaves of this ? What does the decision
tree for MERGESORTING
tree are all possible YES NO four elements look like?
orderings of the
items: when we ? ?
reach a leaf we
YES NO YES NO
return it.
Ollie the

? ? ? ? over-achieving ostrich

Running the algorithm on a given

input corresponds to taking a
particular path through the tree.
What’s the runtime on a particular input?
At least the number
of comparisons that If we take this path through
are made on that ? the tree, the runtime is
input. Ω(length of the path).
YES NO

? ?
YES NO YES NO

? ? ? ?
What’s the worst-case runtime?
At least Ω(length of the longest path).
?
YES NO

? ?
YES NO YES NO

? ? ? ?
 being sloppy about
floors and ceilings!

How long is the longest path?

We want a statement: in all such trees,
the longest path is at least _____

? • This is a binary tree with at

YES NO
n!
least _____ leaves.
? ?
YES
NO YES
NO
• The shallowest tree with n!
? ? ? ? leaves is the completely
balanced one, which has
log(n!)
depth ______.

• So in all such trees, the

longest path is at least log(n!).
• n! is about (n/e)n (Stirling’s approx.*). Conclusion: the longest path
• log(n!) is about n log(n/e) = Ω(n log(n)). has length at least Ω(n log(n)).
*Stirling’s approximation is a bit more complicated than this, but this is good enough for the asymptotic result we want.
 Lower bound of Ω(n log(n)).
• Theorem:
• Any deterministic comparison-based sorting algorithm must
take Ω(n log(n)) steps.

• Proof:
• Any deterministic comparison-based algorithm can be
represented as a decision tree with n! leaves.
• The worst-case running time is at least the depth of the decision
tree.
• All decision trees with n! leaves have depth Ω(n log(n)).

• So any comparison-based sorting algorithm must have worst-

case running time at least Ω(n log(n)).
Aside:
What about randomized algorithms?
• For example, QuickSort?
• Theorem:
• Any randomized comparison-based sorting algorithm
must take Ω(n log(n)) steps in expectation.

• Proof:
• see lecture notes
• (same ideas as deterministic case) Try to prove this
yourself!

\end{Aside} Ollie the over-achieving ostrich

So, MergeSort is optimal!
• This is one of the cool things about lower bounds like
this: we know when we can declare victory!

But what about StickSort?

• StickSort can’t be implemented as a comparison-based
sorting algorithm. So these lower bounds don’t apply.
• But StickSort was kind of dumb.
Especially if I have
to spend time
cutting all those
But might there be another model sticks to be the
right size!
of computation that’s less dumb,
in which we can sort faster?
Beyond comparison-based
sorting algorithms
Another model of computation
• The items you are sorting have meaningful values.

9 6 3 5 2 1 2

instead of
Pre-lecture exercise
• Sorting CS161 students by their month of birth.
• [Discussion on board]

1 1 4 5
Another model of computation
• The items you are sorting have meaningful values.

9 6 3 5 2 1 2

instead of
 Implement the buckets as linked
Why might this help? lists. They are first-in, first-out.
This will be useful later.

BucketSort: 9 6 3 5 2 1 2
Note: this is a simplification of
what CLRS calls “BucketSort”

2
1 2 3 5 6 9
1 2 3 4 5 6 7 8 9

Concatenate SORTED!
the buckets!
In time O(n).
Issues
• Need to be able to know what bucket to put something in.
• That’s okay for now: it’s part of the model.
• Need to know what values might show up ahead of time.

2 12345 13 21000 50 100000000 1

One solution: RadixSort
• Idea: BucketSort on the least-significant digit first,
then the next least-significant, and so on.

Step 1: BucketSort on LSB:

21 345 13 101 50 234 1

345
234
101
50 21 13
0 1 2 3 4 5 6 7 8 9

50 21 101 1 13 234 345

Step 2: BucketSort on the 2nd digit

50 21 101 1 13 234 345

1
345
234

101 13 21 50

0 1 2 3 4 5 6 7 8 9

101 1 13 21 234 345 50

Step 3: BucketSort on the 3rd digit
101 1 13 21 234 345 50

50
21
13
234

345

1 101

0 1 2 3 4 5 6 7 8 9

1 13 21 50 101 234 345

It worked!!
 Why does this work?
Original array:

21 345 13 101 50 234 1

Next array is sorted by the first digit.

50 21 101 1 13 234 345

Next array is sorted by the first two digits.

101 1
01 13 21 234 345 50
Next array is sorted by all three digits.

1
001 13
013 21
021 50
050 101 234 345
Sorted array
 Or at least a
little formally!

Formally…
Lucky the lackadaisical lemur

• Argue by induction.
• Inductive hypothesis:
 Why does this work?
Original array:

21 345 13 101 50 234 1

Next array is sorted by the first digit.

50 21 101 1 13 234 345

Next array is sorted by the first two digits.

101 1
01 13 21 234 345 50
Next array is sorted by all three digits.

1
001 13
013 21
021 50
050 101 234 345
Sorted array
 Or at least a
little formally!

Formally…
Lucky the lackadaisical lemur

• Argue by induction.
• Inductive hypothesis:
• After the k’th iteration, the array is sorted by the first k
least-significant digits.
• Base case:
• “Sorted by 0 least-significant digits” means not sorted.
• Inductive step: This needs to use: (1) bucket sort
works, and (2) we treat each bucket
• (See lecture notes or CLRS) as a FIFO queue.*

• Conclusion: Plucky the pedantic penguin

• After the d’th iteration, the array is sorted by the d least-

significant digits. Aka, it’s sorted. *the buzzword here is that
bucketSort is stable.
 What is the running time?
The “10” is because we
• Say they are d-digit numbers. are working base 10.
• There are d iterations.
• Each iteration takes time O(n + 10) = O(n)
• Total time: O(nd).
• Say the biggest integer is M. What is d?
• d = log%& 𝑀 + 1
• so O(nd) = O( n log10(M) ).

Can we do better?
what if M = n?
Trade-offs…
• RadixSort works with any base.
• Before we did it base r=10.
• But we could do it base r=2 or r=20 just as easily.
• [On board]

• Running time for general r and M?

• [On board]
 Running time: 𝑂 𝑛 ⋅ log . 𝑀
Trade-offs ctd…
• There are n numbers, biggest one is M.
• What should we choose for r (in terms of M,n)?
There’s some sweet spot… (and maybe it’s growing with M and n?)

IPython Notebook for Lecture 6

We get…
• [Discussion on board…]

Choosing r = n
is pretty good.
What’s the optimal
• If we choose r = n, running time is choice of r?
𝑻 𝒏 = 𝑶 𝒏 ⋅ 𝐥𝐨𝐠 𝐧 𝑴
• If M = O(n), T(n) = O(n). Awesome!
• If M = Ω(nn), T(n) = O(n2)…

Ollie the over-achieving ostrich

The story so far
• If we use a comparison-based sorting algorithm, it
MUST run in time Ω(nlog(n)).

• If we assume a bit of structure on the values, we

have an O(n)-time sorting algorithm.

9 6 3 5 2 1 2

Why would we ever use a

comparison-based sorting algorithm??
Why would we ever use
a comparison-based sorting algorithm?
Even with integers, if the
• Lots of precision… biggest on is really big,
RadixSort is slow.
123456
𝜋 987654 𝑒 140! 2.1234123 nn 42
• We can compare these pretty quickly (just look at the most-significant digit):
• 𝜋 = 3.14….
• e = 2.78….
• But to do RadixSort we’d have to look at every digit.
• This is especially problematic since both of these have infinitely many digits...

• RadixSort needs extra memory for the buckets.

• Not in-place
• I want to sort emoji by talking to a genie.
• RadixSort makes more assumptions on the input.
 Recap
• How difficult a problem is depends on the model of
computation.
• How reasonable a model of computation is is up for debate.

• Comparison-based sorting model

• This includes MergeSort, QuickSort, InsertionSort
• Any algorithm in this model must use at least Ω(n log(n)) operations.

• But if we are sorting small integers (or other reasonable data):

• BucketSort and RadixSort
• Both run in time O(n)
Next time
• Binary search trees!
• Balanced binary search trees!
•

Before next time

• Pre-lecture exercise for Lecture 7
• Remember binary search trees?