Algorithm Efficiency and Big-O
Running Time
2
Most algorithms transform best case
average case
input objects into output worst case
objects. 120
The running time of an 100
algorithm typically grows
Running Time
80
with the input size.
60
Average case time is often
difficult to determine. 40
We focus on the worst case 20
running time. 0
1000 2000 3000 4000
Easier to analyze Input Size
Crucial to applications such as
games, finance and robotics
Analysis of Algorithms
Experimental 9000
Studies 8000
7000
Write a program 6000
Time (ms)
implementing the algorithm 5000
Run the program with 4000
inputs of varying size and 3000
composition, noting the
2000
time needed:
1000
Plot the results
0
0 50 100
Input Size
© 2014 Goodrich,
3 Analysis of Algorithms
Tamassia, Goldwasser
Limitations of Experiments
4
It is necessary to implement the algorithm,
which may be difficult
Results may not be indicative of the running
time on other inputs not included in the
experiment.
In order to compare two algorithms, the same
hardware and software environments must be
used
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Theoretical Analysis
5
Uses a high-level description of the
algorithm instead of an implementation
Characterizes running time as a function of
the input size, n
Takes into account all possible inputs
Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Pseudocode
6
High-level description of an algorithm
More structured than English prose
Less detailed than a program
Preferred notation for describing algorithms
Hides program design issues
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Pseudocode Details
Analysis of
Algorithms
Control flow Method call
if … then … [else …] method (arg [, arg…])
while … do … Return value
repeat … until … return expression
for … do … Expressions:
Indentation replaces braces Assignment
Method declaration = Equality testing
Algorithm method (arg [, arg…])
Input … n2 Superscripts and other
Output … mathematical formatting
allowed
© 2014 Goodrich,
7
Tamassia, Goldwasser
The Random Access Machine (RAM)
Model
8
A RAM consists of
A CPU
An potentially unbounded bank
of memory cells, each of which 2
1
can hold an arbitrary number or 0
character
Memory cells are numbered and
accessing any cell in memory
takes unit time
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Seven Important Functions
9
❑ Seven functions that often
1E+30
appear in algorithm 1E+28 Cubic
analysis: 1E+26
1E+24 Quadratic
◼ Constant 1
1E+22
Linear
◼ Logarithmic log n 1E+20
◼ Linear n 1E+18
T (n )
1E+16
◼ N-Log-N n log n 1E+14
◼ Quadratic n2 1E+12
1E+10
◼ Cubic n3 1E+8
◼ Exponential 2n 1E+6
1E+4
1E+2
❑ In a log-log chart, the slope 1E+0
of the line corresponds to 1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
the growth rate n
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Log-log chart/plot
10
y = xa
log y = a log x
Slope is a
© 2014 Goodrich, Tamassia,
Analysis of Algorithms
Goldwasser
Functions Graphed Slide by Matt Stallmann
included with permission.
Using “Normal” Scale
11
g(n) = n lg n
g(n) = 1 g(n) = 2n
g(n) = n2
g(n) = lg n
g(n) = n
g(n) = n3
© 2014 Goodrich,
Analysis of Algorithms
Tamassia, Goldwasser
Primitive Operations
Analysis of
Algorithms
Basic computations
Examples:
performed by an algorithm
Evaluating an
Identifiable in pseudocode expression
Largely independent from the Assigning a value to
a variable
programming language
Indexing into an
Exact definition not important array
(we will see why later) Calling a method
Assumed to take a constant Returning from a
method
amount of time in the RAM
model
© 2014 Goodrich,
12
Tamassia, Goldwasser
Algorithm Efficiency and Big-O
Getting a precise measure of the performance of
an algorithm is difficult
Big-O notation expresses the performance of an
algorithm as a function of the number of items to be
processed
This permits algorithms to be compared for
efficiency
For more than a certain number of data items, some
problems cannot be solved by any computer
Linear Growth Rate
If processing time increases in proportion to the
number of inputs n, the algorithm grows at a linear
rate
public static int search(int[] x, int target) {
for(int i=0; i < x.length; i++) {
if (x[i]==target)
return i;
}
return -1; // target not found
}
Linear Growth Rate
• If the target is not present, the for loop will
execute x.length times
• If the target is present the for loop will execute
(on average) (x.length + 1)/2 times
If processing time• increases in proportion to the
Therefore, the total execution time is directly
number of inputs n, the algorithm grows at a linear
proportional to x.length
rate •
•
This is described as a growth rate of order n OR
O(n)
public static int search(int[] x, int target) {
for(int i=0; i < x.length; i++) {
if (x[i]==target)
return i;
}
return -1; // target not found
}
n x m Growth Rate
Processing time can be dependent on two different
inputs
public static boolean areDifferent(int[] x, int[] y) {
for(int i=0; i < x.length; i++) {
if (search(y, x[i]) != -1)
return false;
}
return true;
}
n x m Growth Rate (cont.)
• The for loop will execute x.length times
• But it will call search, which will execute
Processing time can be dependent on two different
y.length times
• The total execution time is proportional to
inputs. (x.length * y.length)
• The growth rate has an order of n x m or
• O(n x m)
public static boolean areDifferent(int[] x, int[] y) {
for(int i=0; i < x.length; i++) {
if (search(y, x[i]) != -1)
return false;
}
return true;
}
Quadratic Growth Rate
If processing time is proportional to the square of the
number of inputs n, the algorithm grows at a quadratic rate
public static boolean areUnique(int[] x) {
for(int i=0; i < x.length; i++) {
for(int j=0; j < x.length; j++) {
if (i != j && x[i] == x[j])
return false;
}
}
return true;
}
Quadratic Growth Rate (cont.)
• The for loop with i as index will execute
x.length times
If processing time is •proportional to the
The for loop with square
j as of execute
index will the
number of inputs n, thex.length
algorithm times
grows at a quadratic rate
• The total number of times the inner loop will
execute is (x.length)2
• The growth rate has an order of n2 or
• O(n2)
public static boolean areUnique(int[] x) {
for(int i=0; i < x.length; i++) {
for(int j=0; j < x.length; j++) {
if (i != j && x[i] == x[j])
return false;
}
}
return true;
}
Big-O Notation
The O() in the previous examples can be thought of as
an abbreviation of "order of magnitude"
A simple way to determine the big-O notation of an
algorithm is to look at the loops and to see whether the
loops are nested
Assuming a loop body consists only of simple
statements,
a single loop is O(n)
a pair of nested loops is O(n2)
a nested pair of loops inside another is O(n3)
and so on . . .
Big-O Notation (cont.)
You must also examine the number of times a loop is executed
for(i=1; i < x.length; i *= 2) {
// Do something with x[i]
}
The loop body will execute k-1 times, with i having the
following values:
1, 2, 4, 8, 16, . . ., 2k
until 2k is greater than x.length
Since 2k-1 = x.length < 2k and log22k is k, we know that k-1
= log2(x.length) < k
Thus we say the loop is O(log n) (in analyzing algorithms, we
use logarithms to the base 2)
Logarithmic functions grow slowly as the number of data items
n increases
Formal Definition of Big-O
Consider the following program structure:
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
Simple Statement
}
}
for (int i = 0; i < n; i++) {
Simple Statement 1
Simple Statement 2
Simple Statement 3
Simple Statement 4
Simple Statement 5
}
Simple Statement 6
Simple Statement 7
...
Simple Statement 30
Formal Definition of Big-O (cont.)
Consider the following program structure:
for (int i = 0; i < n; i++) { This nested loop
for (int j = 0; j < n; j++) { executes a Simple
Simple Statement Statement n2 times
}
}
for (int i = 0; i < n; i++) {
Simple Statement 1
Simple Statement 2
Simple Statement 3
Simple Statement 4
Simple Statement 5
}
Simple Statement 6
Simple Statement 7
...
Simple Statement 30
Formal Definition of Big-O (cont.)
Consider the following program structure:
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
Simple Statement
}
}
for (int i = 0; i < n; i++) {
Simple Statement 1
Simple Statement 2
This loop executes 5
Simple Statement 3 Simple Statements n times
Simple Statement 4 (5n)
Simple Statement 5
}
Simple Statement 6
Simple Statement 7
...
Simple Statement 30
Formal Definition of Big-O (cont.)
Consider the following program structure:
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
Simple Statement
}
}
for (int i = 0; i < n; i++) {
Simple Statement 1
Simple Statement 2
Simple Statement 3
Simple Statement 4
Simple Statement 5
} Finally, 25 Simple
Simple Statement 6 Statements are executed
Simple Statement 7
...
Simple Statement 30
Formal Definition of Big-O (cont.)
Consider the following program structure:
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
Simple Statement
} We can conclude that the relationship
} between processing time and n (the
for (int i = 0; i < n; i++) { number of date items processed) is:
Simple Statement 1
Simple Statement 2
Simple Statement 3 T(n) = n2 + 5n + 25
Simple Statement 4
Simple Statement 5
}
Simple Statement 6
Simple Statement 7
...
Simple Statement 30
Formal Definition of Big-O (cont.)
In terms of T(n),
T(n) = O(f(n))
There exist
two constants, n0 and c, greater than zero, and
a function, f(n),
such that for all n > n0, cf(n) = T(n)
In other words, as n gets sufficiently large (larger
than n0), there is some constant c for which the
processing time will always be less than or equal to
cf(n)
cf(n) is an upper bound on performance
Formal Definition of Big-O (cont.)
The growth rate of f(n) will be determined by the
fastest growing term, which is the one with the
largest exponent
In the example, an algorithm of
O(n2 + 5n + 25)
is more simply expressed as
O(n2)
In general, it is safe to ignore all constants and to
drop the lower-order terms when determining the
order of magnitude
Big-O Example 1
Given T(n) = n2 + 5n + 25, show that this is O(n2)
Find constants n0 and c so that, for all n > n0, cn2 >
n2 + 5n + 25
Find the point where cn2 = n2 + 5n + 25
Let n = n0, and solve for c
c = 1 + 5/ n0, + 25/ n0 2
When n0 is 5(1 + 5/5 + 25/25), c is 3
So, 3n2 > n2 + 5n + 25 for all n > 5
Other values of n0 and c also work
Big-O Example 1 (cont.)
Big-O Example 2
Consider the following loop
for (int i = 0; i < n; i++) {
for (int j = i + 1; j < n; j++) {
3 simple statements
}
}
T(n) = 3(n – 1) + 3 (n – 2) + … + 3
Factoring out the 3,
3(n – 1 + n – 2 + n – 3 + … + 1)
1 + 2 + … + n – 1 = (n x (n-1))/2
Big-O Example 2 (cont.)
Therefore T(n) = 1.5n2 – 1.5n
When n = 0, the polynomial has the value 0
For values of n > 1, 1.5n2 > 1.5n2 – 1.5n
Therefore T(n) is O(n2) when n0 is 1 and c is 1.5
Big-O Example 2 (cont.)
Symbols Used in Quantifying
Performance
Common Growth Rates
Different Growth Rates
Effects of Different Growth Rates
Algorithms with Exponential and
Factorial Growth Rates
Algorithms with exponential and factorial growth
rates have an effective practical limit on the size of
the problem they can be used to solve
With an O(2n) algorithm, if 100 inputs takes an
hour then,
101 inputs will take 2 hours
105 inputs will take 32 hours
114 inputs will take 16,384 hours (almost 2 years!)
Algorithms with Exponential and
Factorial Growth Rates (cont.)
Encryption algorithms take advantage of this
characteristic
Some cryptographic algorithms can be broken in
O(2n) time, where n is the number of bits in the key
A key length of 40 is considered breakable by a
modern computer,
but a key length of 100 bits will take a billion-
billion (1018) times longer than a key length of 40
