Iteration:

Sorting, Scalability, Big O Notation

1
Announcements

¤ Yesterday?
¤ Lab 4

¤ Tonight
¤ Lab 5

¤ Tomorrow
¤ PS 4
¤ PA 4
Yesterday

¤ Quick Review: Sieve of Eratosthenes

¤ Character Comparisons (Unicode)

¤ Linear Search

¤ Sorting
Today

¤ Review: Insertion Sort

¤ Scalability

¤ Big O Notation
Sorting

5
In-place Insertion Sort

¤ Idea: during sorting, a prefix of the list is already sorted. (This

prefix might contain one, two, or more elements.)

¤ Each element that we process is inserted into the correct

place in the sorted prefix of the list.

¤ Result: sorted part of the list gets bigger until the whole
thing is sorted.
 In-place Insertion Sort

sorted part
 In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part

15110 Principles of Computing 9

Carnegie Mellon University
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
In-place Insertion Sort

sorted part
 In-place Insertion Sort Algorithm
Given a list a of length n, n > 0.

1. Set i = 1.

2. While i is not equal to n, do the following:

a. Insert a[i] into its correct position in a[0] to a[i] (inclusive).

b. Add 1 to i.

3. Return the list a (which is now sorted).

 Example

a = [53, 26, 76, 30, 14, 91, 68, 42]

i = 1

Insert a[1] into its correct position in a[0..1]

and then add 1 to i:
53 moves to the right,
26 is inserted into the list at position 0
a = [26, 53, 76, 30, 14, 91, 68, 42]
i = 2
 Writing the Python code
def isort(items):

i = 1

while i < len(items):

move_left(items, i)
insert a[i] into a[0..i]
i = i + 1 in its correct sorted
position
return items
 Moving left using search
To move the element x at index i “left” to its
correct position, remove it, start at position i-
1, and search from right to left until we find the
first element that is less than or equal to x.
Then insert x back into the list to the right of
that element.
(The Python insert operation does not
overwrite. Think of it as “squeezing into the
list”.)
 Moving left (numbers)
76:

a = [26, 53, 76, 30, 14, 91, 68, 42]

Searching from right to left starting with 53, the first element less than 76 is 53.
Insert 76 to the right of 53 (where it was before).

14:

a = [26, 30, 53, 76, 14, 91, 68, 42]

Searching from right to left starting with 76, all elements left of 14 are greater
than 14. Insert 14 into position 0.

68:

a = [14, 26, 30, 53, 76, 91, 68, 42]

Searching from right to left starting with 91, the first element less than 68 is 53.

Insert 68 to the right of 53.

 The move_left algorithm

Given a list a of length n, n > 0 and a value at

index i to be moved left in the list.
1. Remove a[i] from the list and store in x.
2. Set j = i-1.
3. While j >= 0 and a[j] > x, subtract 1 from j.
4. (At this point, what do we know? Either j is …,
or a[j] is …) Insert x into position a[j+1].
 Removing a list element: pop
>>> a = ["Wednesday", "Monday", "Tuesday"]
>>> day = a.pop(1)
>>> a
['Wednesday', 'Tuesday']
>>> day
'Monday'
>>> day = a.pop(0)
>>> day
'Wednesday'
>>> a
['Tuesday']
Inserting an element: insert

>> a = [10, 20, 30]

=> [10, 20, 30]
>> a.insert(0, "foo")
=> ["foo", 10, 20, 30]
>> a.insert(2, "bar")
=> ["foo", 10, "bar", 20, 30]
>> a.insert(5, "baz")
=> ["foo", 10, "bar", 20, 30, "baz"]
 move_left in Python

def move_left(items, i):

x = items.pop(i)

j = i - 1

while j >= 0 and items[j] > x:

j = j – 1

items.insert(j + 1, x)
Problems, Algorithms and Programs

¤ One problem : potentially many algorithms

¤ One algorithm : potentially many programs

¤ We can compare how efficient different programs are

both analytically and empirically

31
 Analytically: Which One is Faster?

def contains1(items, key): def contains2(items, key):

ln = len(items)
index = 0
index = 0
while index < len(items):
while index < ln:
if items[index] == key:
if items[index] == key:
return True
return True
index = index + 1
index = index + 1
return False
return False
¤ len(items) is executed each
len(items) is executed only
time loop condition is checked
once and its value is stored in ln
32
 Is a for-loop faster than a while-loop?

•Add the following function to our collection of

contains functions from the previous page:

def contains3(items, key):

for index in range(len(items)):
if items[index] == key:
return True
return False

33
 Empirical Measurement
¤ Three programs for the same algorithm; let’s measure which is faster:

¤ Define time2 and time3 similarly to call contains2 and contains

import time
def time1(items, key) :
start = time.time()
contains1(items, key)
runtime = time.time() - start
print("contains1:", runtime)

34
 Doing the measurement
>>> items = [None] * 1000000

>>> time1(items1, 1)

contains1: 0.1731700897216797

>>> time2(items1, 1)

contains2: 0.1145467758178711

>>> time3(items1, 1)

contains3: 0.07184195518493652

Conclusion: using for and range() is faster than using

while and addition when doing an unsuccessful
search Why?

35
 A Different Measurement
¤ What if we want to know how the different loops perform when the
key matches the first element?

>>> time1(items1, None)

contains1: 4.0531158447265625e-06

>>> time2(items1, None)

contains2: 4.291534423828125e-06

>>> time3(items1, None)

contains3: 1.0013580322265625e-05

Now the relationship is different; contains3 is

slowest! Why?

36
Thinking like a computer scientist
Code Analysis
 Efficiency
¤ A computer program should be correct, but it should
also
¤ execute as quickly as possible (time-efficiency)
¤ use memory wisely (storage-efficiency)
¤ How do we compare programs (or algorithms in
general) with respect to execution time?
¤ various computers run at different speeds due to different
processors
¤ compilers optimize code before execution
¤ the same algorithm can be written differently depending
on the programming paradigm
 Counting Operations
¤ We measure time efficiency by considering “work”
done
¤ Counting the number of operations performed by the
algorithm.

¤ But what is an “operation”? Think of it in a

machine-independent way
¤ assignment statements
¤ comparisons
¤ function calls
¤ return statements
¤ We think of an operation as any computation that is
independent of the size of our input.
 Linear Search
# let n = the length of list.
def search(list, key):
index = 0
while index < len(list):
if list[index] == key:
return index
index = index + 1 Best case: the key is the first
element in the list
return None
 Linear Search: Best Case

# let n = the length of list.

def search(list, key):
index = 0 1
while index < len(list): 1
if list[index] == key: 1
return index 1
index = index + 1
return None
Total:4
 Linear Search: Worst Case
# let n = the length of list.
def search(list, key):
index = 0
while index < len(list):
if list[index] == key:
return index
index = index + 1 Worst case: the key is not an
element in the list
return None
 Linear Search: Worst Case
# let n = the length of list.
def search(list, key):
index = 0 1
while index < len(list): n+1
if list[index] == key: n
return index
index = index + 1 n
return None 1
Total: 3n+3
 Asymptotic Analysis
¤How do we know that each operation we count takes
the same amount of time?
¤ We don’t.
¤So generally, we look at the process more abstractly
¤ We care about the behavior of a program in the long run (on
large input sizes)
¤ We don’t care about constant factors (we care about how
many iterations we make, not how many operations we have
to do in each iteration)
 What Do We Gain?
¤Show important characteristics in terms of resource
requirements
¤Suppress tedious details
¤Matches the outcomes in practice quite well
¤As long as operations are faster than some constant
(1 ns? 1 μs? 1 year?), it does not matter
 Linear Search: Best Case
Simplified
# let n = the length of list.
def search(list, key):
index = 0
while index < len(list): 1 iteration
if list[index] == key:
return index
index = index + 1
return None
 Linear Search: Worst Case
Simplified
# let n = the length of list.
def search(list, key):
index = 0
while index < len(list): n iterations
if list[index] == key:
return index
index = index + 1
return None
 Order of Complexity
¤ For very large n, we express the number of operations
as the (time) order of complexity.
¤ For asymptotic upper bound, order of complexity is
often expressed using Big-O notation:
¤ Number of operations Order of Complexity
¤ n O(n)
Usually doesn't
¤ 3n+3 O(n) matter what the
constants are...
¤ 2n+8 O(n) we are only
concerned about
the highest power
of n.
Why don’t constants matter?

(n=1) 45n3 + 20n2 + 19 = 84

(n=2) 45n3 + 20n2 + 19 = 459

(n=3) 45n3 + 20n2 + 19 = 1414

 O(n) (“Linear”)
2n + 8
3n+3 n

Number of
Operations

n
(amount of data)
 O(n)
Number of
n
Operations
30 For a linear algorithm,
if you double the amount
of data, the amount of work
you do doubles
20 (approximately).

10

10 20 30 n
(amount of data)
 O(1) (“Constant-Time”)
Number of For a constant-time algorithm,
Operations if you double the amount
of data, the amount of work
you do stays the same.

4 = O(1)
4

1 = O(1)
1

n
(amount of data)
 Linear Search
¤Best Case: O(1)

¤Worst Case: O(n)

¤Average Case: ?
¤ Depends on the distribution of queries
¤ But can’t be worse than O(n)
Insertion Sort
 Insertion Sort

# let n = the length of list.

def isort(list):
i = 1
while i != len(list): n-1 iterations
move_left(list, i)
i = i + 1
return list
 move_left
# let n = the length of list.
def move_left(a, i):
x = a.pop(i)
j = i - 1
while j >= 0 and a[j] > x:
j = j – 1
a.insert(j + 1, x)
 move_left
# let n = the length of list.
def move_left(a, i): at most
x = a.pop(i)
j = i - 1
while j >= 0 and a[j] > x: i iterations
j = j – 1
a.insert(j + 1, x)

but how long do pop and insert take?

 Measuring pop and insert
2 million elements in list, 1000 inserts: 0.7548720836639404 seconds

4 million elements in list, 1000 inserts: 1.6343820095062256 seconds

8 million elements in list, 1000 inserts: 3.327040195465088 seconds

8 million elements in list, 1000 pops: 2.031071901321411 seconds

16 million elements in list, 1000 pops: 4.033380031585693 seconds

32 million elements in list, 1000 pops: 8.06456995010376 seconds

Doubling the size of the list doubles the cost (time) of insert or pop. These functions
take linear time.
 Insertion Sort: cost of move left
# let n = the length of list.
def move_left(a, i):
x = a.pop(i) n iterations
j = i - 1
while j >= 0 and a[j] > x: i iterations
j = j – 1
a.insert(j + 1, x) n iterations

Total cost (at most): n + i + n

But what is i? To find out, look at isort, which calls
move_left, supplying a value for i
 Insertion Sort: what is the cost of
the whole thing?
# let n = the length of list.
def isort(list):
i = 1
while i != len(list): #n-1 iterations
move_left(list,i) #i goes from 1 to n-1
i = i + 1
return list
Total cost: cost of move_left as i goes from 1 to n-1

Cost of all the move_lefts: n + 1 + n

+n+ 2 + n
+n+ 3 + n
...
+n+ n-1 + n
 In place iSort Worst Case…

j i

¤ • On iteration i, we need to examine j elements and then

shift i-j elements to the right, so we have to do j + (i-j) = i
units of work.
 Remember…

¤ What are we trying to do?

¤ Understand the cost of insertion sort
¤ How do we understand that cost?
¤ Via order of complexity – finding the highest order term
¤ What does this require?
¤ 1st generalizing the cost as an equation
¤ Then simplifying the equation to find highest order term
 Figuring out the sum
¤ n+1+n (n-1)*2n
¤ +n+2+n +1
¤ +n+3+n
+2
¤ ...
+3
¤ + n + n-1 + n
...
+ n-1
 Adding 1 through n-1
1 2 3 4 5 6 7

2
(6 * 7) / 2
3 blue circles

6
 Adding 1 through n-1
¤ We saw 1 + 2 + ... + 6 = (6 * 7) / 2

¤ Generalizing, 1 + 2 + ... + n-1 = (n-1)(n) / 2

¤ So our whole cost is:

¤ (n-1)*2n + 1 + 2 + 3 ... + n-1
¤ = (n-1)*2n + (n-1)(n) / 2
¤ = 2n2 - 2n + (n2 - n) / 2
¤ = (5n2 - 5n) / 2 = (5/2)n2 - (5/2)n
¤ Observe that the highest-order term is n2
A different way…

¤ When i=1,we have1 unit of work.

¤ When i=2, we have 2 units of work.

¤ ...
¤ When i = n-1, we have n-1 units of work.
¤ The total amount of work done is:

¤ 1 + 2 + ... + (n-1)
= n(n-1)/2
= (n2 - n)/2 (a quadratic function)
= O(n2)
Let’s look at this again…
slowly
 Examining the cost
def isort(list):
i = 1
while i != len(list):
move_left(list,i)
i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list):
move_left(list,i)
i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop
while loop
insert
i = i + 1
return list
 Again…
2 million elements in list, 1000 inserts: 0.7548720836639404 seconds

4 million elements in list, 1000 inserts: 1.6343820095062256 seconds

8 million elements in list, 1000 inserts: 3.327040195465088 seconds

8 million elements in list, 1000 pops: 2.031071901321411 seconds

16 million elements in list, 1000 pops: 4.033380031585693 seconds

32 million elements in list, 1000 pops: 8.06456995010376 seconds

Doubling the size of the list doubles the cost (time) of insert or pop. These functions
take linear time.
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop…………………..…….. n
while loop
insert
i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop…………………..…….. n
while loop
insert …………………..…. n
i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... n + n
while loop

i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop

i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+

i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+

i = i + 1
return list
 Examining the cost
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+3…

i = i + 1
return list
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+3…n-1

i = i + 1
return list
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+3…n-1

i = i + 1
return list

1+2+3…n-1
Test for n = 7

1+2+3…n-1
Test for n = 7.

1+2+3+4+5+6
1+2+3…n-1
Test for n = 7.

1+2+3+4+5+6
1+2+3…n-1
Test for n = 7.

(6) * (7) / 2 blue circles

1+2+3+4+5+6
1+2+3…n-1
Test for n = 7.

(6) * (7) / 2 blue circles

(n-1) * (n) / 2 blue circles

1+2+3+4+5+6
1+2+3…n-1
Our equation …

(6) * (7) / 2 blue circles

(n-1) * (n) / 2 blue circles

(n-1)*n/2
1+2+3…n-1
Our equation …

(n-1)*n/2
1+2+3…n-1
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+3…n-1

(n-1)*n/2i = i + 1
return list

1+2+3…n-1
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n
while loop 1+2+3…n-1

i = i + 1
return list

(n-1)*n/2
 Combine to calculate
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert ......... 2n +
while loop (n-1)*n/2

i = i + 1
return list
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i)
pop & insert & while 2n + (n-1)*n/2

i = i + 1
return list
 How can we express this?
def isort(list):
i = 1
while i != len(list): n-1
move_left(list,i) 2n + (n-1)*n/2

i = i + 1
return list
 Total number of operations
def isort(list):
i = 1
while i != len(list): (n-1) * 2n + (n-1)*n/2
move_left(list,i)

i = i + 1
return list
 Generalizing…

(n-1) * 2n + (n-1)*n/2

¤ = 2n2 - 2n + (n2 - n) / 2
¤ = (5n2 - 5n) / 2
¤ = (5/2)n2 - (5/2)n
Highest order term? …

(5/2)n2 – (5/2)n

n2
 Order of Complexity

Number of operations Order of Complexity

n2 O(n2)
(5/2)n2 - (1/2)n O(n2)
2n2 + 7 O(n2)
Usually doesn’t matter what
the constants are… f(n) is O(g(n)) means
we are only concerned about
the highest power of n.
f(n) < g(n)･k for some
positive k
 Keep It Simple
¤ “Big O” notation expresses an upper bound:
f(n) is O(g(n)) means f(n) < g(n)･k
(whenever n is large enough)
¤ So if f(x) is O(n2), then f(x) is O(n3) too!
¤ But we always use the smallest possible function, and the
simplest possible.
¤ We say 3n2 + 4n + 1 is O(n2), not O(n3)
¤ We say 3n2 + 4n + 1 is O(n2), not O(3n2 + 4n)
¤ ...even though all of the above are true
 O(n2) (“Quadratic”)
n2
2n2 + 7 n2/2 + 3n/2 – 1

Number of
Operations

n
(amount of data)
 O(n2)
Number of
N2
Operations
For a quadratic algorithm,
900 if you double the amount
of data, the amount of work
you do quadruples
(approximately).

400

100

10 20 30 N
(amount of data)
 Insertion Sort
¤ Worst Case: O(n2)

¤ Best Case: ?

¤ Average Case: ?

¤ We’ll compare these algorithms with others soon to see how

scalable they really are based on their order of complexities.
 Big O

¤ O(1) constant

¤ O(log n) logarithmic

¤ O(n) linear

¤ O(n log n) log linear

¤ O(n2) quadratic

¤ O(n3) cubic

¤ O(2n) exponential