Sorting
EECS 2011
www.eecs.yorku.ca/course_archive/2017-18/W/2011MF/
1/95
Overview
Review
Lower bound
Linear sorting
2/95
Sorting algorithms
Question
http://www.eecs.yorku.ca/course_archive/2017-18/W/
2011MF/sorting.html
3/95
Sorting algorithms
Answer
1 Merge sort
2 Heap sort
3 Selection sort
4 Quick sort
5 Bubble sort
6 Insertion sort
4/95
Big O, Big Omega and Big Theta
To capture the running time of algorithms, we use the following
notation. Let f , g ∈ N → N.
f ∈ O(g ) if
∃k > 0 : ∃m ∈ N : ∀n ≥ m : f (n) ≤ k g (n).
f is bounded from above by g
f ∈ Ω(g ) if
∃k > 0 : ∃m ∈ N : ∀n ≥ m : k g (n) ≤ f (n).
f is bounded from below by g
f ∈ Θ(g ) if
∃k` > 0 : ∃ku > 0 : ∃m ∈ N : ∀n ≥ m : k` g (n) ≤ f (n) ≤ ku g (n).
f is bounded from above and below by g
5/95
Sorting algorithms
selection sort Θ(n2 )
insertion sort Θ(n2 )
bubble sort Θ(n2 )
merge sort Θ(n log n)
quick sort Θ(n2 )
heap sort Θ(n log n)
6/95
Overview
Review
Lower bound
Linear sorting
7/95
How fast can we sort?
Question
How fast can we sort?
8/95
How fast can we sort?
Question
How fast can we sort?
Answer
We can sort n elements in Θ(n log n).
8/95
How fast can we sort?
Question
How fast can we sort?
Answer
We can sort n elements in Θ(n log n).
Question
Can we sort any faster?
8/95
How fast can we sort?
Question
How fast can we sort?
Answer
We can sort n elements in Θ(n log n).
Question
Can we sort any faster?
Question
Does there exist an algorithm that can sort n elements any faster?
8/95
Comparison sorts
Definition
A sorting algorithm is a comparison sort if the sorting is based on
comparisons between the elements (and not on the values of the
elements).
9/95
Comparison sorts
insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && a[j - 1] > key) {
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}
Question
Is insertion sort a comparison sort?
10/95
Comparison sorts
insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && a[j - 1] > key) {
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}
Question
Is insertion sort a comparison sort?
Question
Yes.
10/95
Comparison sorts
mergeSort(a, l, u) {
if (l + 1 < u) {
m = (l + u) / 2;
mergeSort(a, l, m);
mergeSort(a, m, u);
merge(a, l, m, u);
}
}
merge(a, l, m, u) {
i = l; j = m;
for (k = l; k < u; k = k + 1) {
if (i < m && (j >= u || a[i] <= a[j])) {
b[k] = a[i]; i = i + 1;
} else {
b[k] = a[j]; j = j + 1;
}
}
for (k = l, k < u; k = k + 1) {
a[k] = b[k];
} 11/95
Comparison sorts
Question
Is merge sort a comparison sort?
12/95
Comparison sorts
Question
Is merge sort a comparison sort?
Question
Yes.
12/95
Comparison sorts
Question
Is merge sort a comparison sort?
Question
Yes.
We will see examples of sorting algorithms that are not comparison
sorts later in this lecture.
12/95
Lower bound
Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.
13/95
Lower bound
Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.
Corollary
The worst case running time of any comparison sort is Ω(n log n).
13/95
Lower bound
Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.
Corollary
The worst case running time of any comparison sort is Ω(n log n).
Corollary
Merge sort and heap sort are asymptotically optimal comparison
sorts.
13/95
Proof of theorem
Without loss of generality, we may assume that all elements to be
sorted are different.
Question
Why can we assume that?
14/95
Proof of theorem
Without loss of generality, we may assume that all elements to be
sorted are different.
Question
Why can we assume that?
Answer
Assume that some elements are the same.
1, 6, 1, 6, 6
Add fractions to make them all different (in linear time).
1, 6, 1 12 , 6 13 , 6 23
Sort them.
1, 1 12 , 6, 6 13 , 6 23
Drop fractions (in linear time).
1, 1, 6, 6, 6
14/95
Comparisons
Given two elements ai and aj , we can compare then using
ai < aj
ai ≤ aj
ai = aj
ai ≥ aj
ai > aj
15/95
Comparisons
Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?
16/95
Comparisons
Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?
Answer
The other comparators <, ≥ and > can all be expressed in terms
of ≤ since
ai < aj iff ai ≤ aj
iff aj ≥ ai
iff aj > ai
16/95
Comparisons
Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?
Answer
The other comparators <, ≥ and > can all be expressed in terms
of ≤ since
ai < aj iff ai ≤ aj
iff aj ≥ ai
iff aj > ai
Corollary
If all elements are different, then each comparison sort algorithm
can be rewritten so it only uses ≤.
16/95
Decision tree
Definition
Given the number of elements to be sorted, the decision tree for a
sorting algorithm is a binary tree containing the comparisons
performed by the sorting algorithm.
17/95
Decision tree
insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && key <= a[j - 1]) { // use <=
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}
18/95
Decision tree
The decision tree for insertion sort for three elements can be
depicted as follows.
a1 ≤ a0
a2 ≤ a0 a2 ≤ a1
a2 ≤ a1 h1, 0, 2i a2 ≤ a0 h0, 1, 2i
h2, 1, 0i h1, 2, 0i h2, 0, 1i h0, 2, 1i
19/95
Decision tree
selectionSort(a) {
for (i = 0; i < a.length; i = i + 1) {
min = i;
for (j = i + 1; j < a.length; j = j + 1) {
if (a[j] <= a[min]) { // use <=
min = j;
}
}
temp = a[i];
a[i] = a[min];
a[min] = temp;
}
}
20/95
Decision tree
Question
Draw the decision tree for selection sort for three elements.
21/95
Decision tree
Answer
a1 ≤ a0
a2 ≤ a1 a2 ≤ a0
a0 ≤ a1 a2 ≤ a0 a1 ≤ a0 a1 ≤ a2
h2, 1, 0i h1, 2, 0i h1, 0, 2i h2, 0, 1i h0, 1, 2i h0, 2, 1i
22/95
Height of decision tree
Observation
The worst case running time of a sorting algorithm of n elements
the maximal number of comparisons of a sorting algorithm of n
elements
=
the height of the decision tree for the sorting algorithm of n
elements.
23/95
Decision tree
Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?
24/95
Decision tree
Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?
Answer
Permutations of 0, 1, . . . , n − 1.
24/95
Decision tree
Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?
Answer
Permutations of 0, 1, . . . , n − 1.
Question
How many permutations of 0, 1, . . . , n − 1 are there?
24/95
Decision tree
Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?
Answer
Permutations of 0, 1, . . . , n − 1.
Question
How many permutations of 0, 1, . . . , n − 1 are there?
Answer
n × (n − 1) × · · · 2 × 1 = n!.
24/95
Properties of decision trees
Property (Proposition 8.7 of the textbook)
A binary tree of height h has at most 2h leaves.
25/95
Properties of decision trees
Property (Proposition 8.7 of the textbook)
A binary tree of height h has at most 2h leaves.
Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.
25/95
Properties of decision trees
Property (Proposition 8.7 of the textbook)
A binary tree of height h has at most 2h leaves.
Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.
Question
Why does a decision tree for a sorting algorithm of n elements
have at least n! leaves?
25/95
Properties of decision trees
Property (Proposition 8.7 of the textbook)
A binary tree of height h has at most 2h leaves.
Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.
Question
Why does a decision tree for a sorting algorithm of n elements
have at least n! leaves?
Answer
Because the n elements can be ordered in any order and, hence,
any permutation is a possible outcome.
25/95
Properties of decision trees
Property
A binary tree of height h has at most 2h leaves.
Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.
Conclusion
2h ≥ n! and, hence,
n
h ≥ log(n!) ≥ log ( n2 ) 2 = n
2 log( n2 ) ∈ Ω(n log n).
26/95
Lower bound
Observation
The worst case running time of a sorting algorithm of n elements
the maximal number of comparisons of a sorting algorithm of n
elements
=
the height of the decision tree for the sorting algorithm of n
elements
≥
n
2 log( n2 ) ∈ Ω(n log n).
27/95
Lower bound
Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.
Corollary
The worst case running time of any comparison sort is Ω(n log n).
Corollary
Merge sort and heap sort are asymptotically optimal comparison
sorts.
28/95
Overview
Review
Lower bound
Linear sorting
29/95
But first...
... a break.
30/95
Overview
Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort
31/95
Linear time sorting
Theorem
The worst case running time of any comparison sort is Ω(n log n).
Question
Can we do better if use information other than comparison of
elements?
32/95
Linear time sorting
Theorem
The worst case running time of any comparison sort is Ω(n log n).
Question
Can we do better if use information other than comparison of
elements?
Answer
Yes.
32/95
Overview
Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort
33/95
Bucket sort
Assumption
All elements come from the interval [0, N − 1] for some N ≥ 2.
Main idea
1. Create N buckets.
2. Place each element in “its” bucket.
3. Concatenate the buckets.
34/95
Bucket sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
1. Create five buckets.
0 1 2 3 4
35/95
Bucket sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
2. Place each element in “its” bucket.
0 1 2 3 4
36/95
Bucket sort
Elements to be sorted: 4, 4, 1, 0, 2
2. Place each element in “its” bucket.
0 1 2 3 4
37/95
Bucket sort
Elements to be sorted: 4, 1, 0, 2
2. Place each element in “its” bucket.
0 1 2 3 4
2 4
38/95
Bucket sort
Elements to be sorted: 1, 0, 2
2. Place each element in “its” bucket.
0 1 2 3 4
2 4
39/95
Bucket sort
Elements to be sorted: 0, 2
2. Place each element in “its” bucket.
0 1 2 3 4
1 2 4
40/95
Bucket sort
Elements to be sorted: 2
2. Place each element in “its” bucket.
0 1 2 3 4
0 1 2 4
41/95
Bucket sort
Elements to be sorted:
2. Place each element in “its” bucket.
0 1 2 3 4
0 1 2 4
2 4
42/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
0 1 2 4
2 4
Sorted elements:
43/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
1 2 4
2 4
Sorted elements: 0
44/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
2 4
2 4
Sorted elements: 0, 1
45/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
2 4
Sorted elements: 0, 1, 2
46/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
Sorted elements: 0, 1, 2, 2
47/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
Sorted elements: 0, 1, 2, 2, 4
48/95
Bucket sort
3. Concatenate the buckets.
0 1 2 3 4
Sorted elements: 0, 1, 2, 2, 4, 4
49/95
Bucket sort
Question
How can we represent the buckets?
50/95
Bucket sort
Question
How can we represent the buckets?
Answer
As an array of lists.
50/95
Bucket sort
bucketSort(a, N) {
for (i = 0; i < N; i = i + 1) {
b[i] = empty list;
}
for (i = 0; i < a.length; i = i + 1) {
b[a[i]].add(a[i]);
}
j = 0;
for (i = 0; i < N; i = i + 1) {
while (!b[i].isEmpty()) {
a[j] = b[i].remove();
j++;
}
}
}
51/95
Bucket sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
52/95
Bucket sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
Answer
O(n + N).
52/95
Bucket sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
Answer
O(n + N).
Note
If N ∈ O(n) then the worst case running time of bucket sort is
O(n).
52/95
Overview
Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort
53/95
Counting sort
Assumption
All elements come from the interval [0, N − 1] for some N ≥ 2.
Main idea
1. Create a frequency table with N entries.
2. Keep track of the frequency of each element.
3. Compute the number of elements smaller than or equal to a
given element.
4. Place each element in “right” place.
54/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
1. Create a frequency table with five entries.
0 1 2 3 4
0 0 0 0 0
55/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
2. Keep track of the frequency of each element.
0 1 2 3 4
0 0 0 0 0
56/95
Counting sort
Elements to be sorted: 4, 4, 1, 0, 2
2. Keep track of the frequency of each element.
0 1 2 3 4
0 0 1 0 0
57/95
Counting sort
Elements to be sorted: 4, 1, 0, 2
2. Keep track of the frequency of each element.
0 1 2 3 4
0 0 1 0 1
58/95
Counting sort
Elements to be sorted: 1, 0, 2
2. Keep track of the frequency of each element.
0 1 2 3 4
0 0 1 0 2
59/95
Counting sort
Elements to be sorted: 0, 2
2. Keep track of the frequency of each element.
0 1 2 3 4
0 1 1 0 2
60/95
Counting sort
Elements to be sorted: 2
2. Keep track of the frequency of each element.
0 1 2 3 4
1 1 1 0 2
61/95
Counting sort
Elements to be sorted:
2. Keep track of the frequency of each element.
0 1 2 3 4
1 1 2 0 2
62/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
0 0 0 0 0
63/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
1 0 0 0 0
64/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
1 2 0 0 0
65/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
1 2 4 0 0
66/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 0
67/95
Counting sort
3. Compute the number of elements smaller than or equal to a
given element.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 6
68/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 6
Sorted elements:
69/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 6
Sorted elements:
70/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 6
Sorted elements:
71/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 4 4 6
Sorted elements:
72/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 3 4 6
Sorted elements:
73/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 3 4 6
Sorted elements:
74/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 3 4 6
Sorted elements:
75/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
1 2 3 4 6
Sorted elements:
0 2
76/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 2 3 4 6
Sorted elements:
0 2
77/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 2 3 4 6
Sorted elements:
0 2
78/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 2 3 4 6
Sorted elements:
0 2
79/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 2 3 4 6
Sorted elements:
0 1 2
80/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 6
Sorted elements:
0 1 2
81/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 6
Sorted elements:
0 1 2
82/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 6
Sorted elements:
0 1 2
83/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 6
Sorted elements:
0 1 2 4
84/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 5
Sorted elements:
0 1 2 4
85/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 5
Sorted elements:
0 1 2 4
86/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 5
Sorted elements:
0 1 2 4
87/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 5
Sorted elements:
0 1 2 4 4
88/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 4
Sorted elements:
0 1 2 4 4
89/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 4
Sorted elements:
0 1 2 4 4
90/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 4
Sorted elements:
0 1 2 4 4
91/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 3 4 4
Sorted elements:
0 1 2 2 4 4
92/95
Counting sort
Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.
0 1 2 3 4
1 1 2 0 2
0 1 2 4 4
Sorted elements:
0 1 2 2 4 4
93/95
Counting sort
countingSort(a, N) {
for (i = 0; i < N; i = i + 1) {
f[i] = 0;;
}
for (i = 0; i < a.length; i = i + 1) {
f[a[i]] = f[a[i]] + 1;
}
for (i = 1; i < N; i = i + 1) {
f[i] = f[i-1] + f[i];
}
for (i = a.length - 1; i >= 0; i = i - 1) {
b[f[a[i]] = a[i];
f[a[i]] = f[a[i]] - 1;
}
for (i = 0; i < a.length; i++) {
a[i] = b[i];
}
94/95
Counting sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
95/95
Counting sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
Answer
O(n + N).
95/95
Counting sort
Question
Express the worst case running time of bucket sort in terms of n
and N.
Answer
O(n + N).
Note
If N ∈ O(n) then the worst case running time of bucket sort is
O(n).
95/95
Radix Sort
Input:
• An array of N numbers.
• Each number contains d digits.
• Each digit between [0…k-1]
Output:
• Sorted numbers.
Each digit (column) can be sorted (e.g., using
Counting Sort).
Which digit to start from?
RadixSort
344 125 125
125 134 224
333 Sort by which 143 Sort by which 225
134 digit first? 224 digit Second? 325
224 225 134
334 The most 243 The next most 333
143 significant. 344 significant. 334
225 333 143
325 334 243
243 325 344
All meaning in first sort lost.
Radix Sort
1. Start from the least significant digit, sort
2. Sort by the next least significant digit
3. Are the last 2 columns sorted?
4. Generalize: after j iterations, the last j columns
are sorted
5. Loop invariant: Before iteration i, the keys
have been correctly stable-sorted with
respect to the i-1 least-significant digits.
Radix sort
Radix-Sort(A,d)
• for i←1 to d
• do use a stable sort to sort A on digit i
Analysis:
Given n d-digit numbers where each digit takes on
up to k values, Radix-Sort sorts these numbers
correctly in Θ(d(n+k)) time.
Radix sort – example (binary)
Sorting a sequence of 4-bit integers
1001 0010 1001 1001 0001
0010 1110 1101 0001 0010
1101 1001 0001 0010 1001
0001 1101 0010 1101 1101
1110 0001 1110 1110 1110
