Algorithm Analysis and
Asymptotic Notations
Data Structures Department of Computer Science – University of Zakho 1
What is the algorithm
An algorithm is a step-by-step procedure for performing some task
(ex: sorting a set of integers) in a finite amount of time.
Input Algorithm Output
We are concerned with the following properties:
Correctness
Efficiency (how fast it is, how many resources it needs)
Data Structures Department of Computer Science – University of Zakho 2
Running Time
Running time is a natural measure of Efficiency.
So what would be the proper way of measuring it?
Do experiments, and then find the run time.
If we have two algorithms for a problem, implement them and do
several experiments on various input size.
Then decide which algorithm is better.
Data Structures Department of Computer Science – University of Zakho 3
Experimental Studies
9000
8000
7000
Run the program with inputs of
6000
varying size and composition
Time (ms)
5000
Use a method like std::clock() to get 4000
an accurate measure of the actual 3000
running time 2000
1000
Plot the results 0
0 50 100
Input Size
Department of Computer Science – University of Zakho 4
What is the problem of experimental studies?
The running time is affected by the hardware (Processor, RAM, etc.)
and software (Compiler, programing language, etc.)
Data Structures Department of Computer Science – University of Zakho 5
Limitations of Experiments
Need to implement the algorithm
may be difficult
Experiments done on a limited set of test inputs
may not be indicative of running times on other inputs not included in the
experiment.
Difficult to compare
same hardware and software environments must be used
Data Structures Department of Computer Science – University of Zakho 6
Running time
We need another way to measure to the running time of an
algorithm which:
Considers all possible inputs.
Be independent from hardware and software.
Data Structures Department of Computer Science – University of Zakho 7
Running Time
The running time of an algorithm typically best case
average case
grows with the input size. worst case
120
100
Average case time is often difficult to
Running Time
80
determine. 60
40
20
We focus on the worst case running time. 0
1000 2000 3000 4000
Easier to analyze Input Size
Crucial to applications such as games, finance,
and robotics
Data Structures Department of Computer Science – University of Zakho 8
Theoretical Analysis
• Uses pseudocode, a high-level description of the algorithm
– no implementation necessary
• Takes into account all possible inputs
• Characterizes running time by f(n), a function of the input size n
– allows us to evaluate the speed of an algorithm independent of
hardware/software environment
Data Structures Department of Computer Science – University of Zakho 9
Pseudocode
Mixture of natural language and high-level
programming constructs that describe the Algorithm arrayMax(A, n)
main ideas behind an algorithm Input array A of n integers
implementation. Output maximum element of A
currentMax A[0]
Preferred notation for
describing algorithms. for i 1 to n 1 do
if A[i] currentMax then
Hides program design issues
currentMax A[i]
return currentMax
Data Structures Department of Computer Science – University of Zakho 10
Pseudocode Details
Control flow Method call
if … then … [else …] var.method (arg [, arg…])
while … do …
Return value
repeat … until …
return expression
for … do …
Indentation replaces braces Expressions
or := Assignment (like in
C++)
Method declaration Equality testing (like in
C++)
Algorithm method (arg [, arg…])
n2 Superscripts and other
Input … mathematical formatting
allowed
Output …
Data Structures Department of Computer Science – University of Zakho 11
The Random Access Machine (RAM) Model
• Views a computer as:
– a CPU, with
– a potentially unbounded bank of 2
memory cells, each of which can 1
0
hold an arbitrary number or
character
Memory cells are numbered and accessing any cell in memory
takes unit time.
Random Access refers to ability of CPU to access arbitrary memory
cell with one primitive operation.
Data Structures Department of Computer Science – University of Zakho 12
Primitive Operations
• Basic computations performed by an algorithm
– Identifiable in pseudocode
– Largely independent from the programming language
– Exact definition not important (we’ll see why later)
• Assumed to take a constant amount of time in the RAM
model
• Includes:
– indexing into an array
– evaluating an expression
– calling a method
– assigning a value to a variable
– returning from a method
Data Structures Department of Computer Science – University of Zakho 13
Counting Primitive Operations
By inspecting the Pseudocode, we can determine the maximum
number of primitive operations executed by an algorithm, as a
function of the input size.
Algorithm arrayMax(A, n) Operations
currentMax A[0] 2
for i 1 to n 1 do 1+n
if A[i] currentMax then 2(n-1)
currentMax A[i] 2(n-1)
{ increment counter i } 2(n-1)
return currentMax 1
An algorithm to find the maximum number in array.
Data Structures Department of Computer Science – University of Zakho 14
Counting Primitive Operations
Analysis of Algorithms 15
Data Structures Department of Computer Science – University of Zakho 15
Estimating Running Time
Algorithm arrayMax executes 7n 2 primitive operations in the worst
case.
Define:
a = time taken by the fastest primitive operation
b = time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax. Then
a(7n 2) T(n) b(7n 2)
Hence, the running time T(n) is bounded by two linear functions.
Data Structures Department of Computer Science – University of Zakho 16
Growth Rate of Running Time
Changing the hardware/software environment
affects T(n) by a constant factor, but
does not alter the growth rate of T(n)
The linear growth rate of the running time T(n) is an intrinsic property
of algorithm arrayMax.
Data Structures Department of Computer Science – University of Zakho 17
Mathematical Review
Data Structures Department of Computer Science – University of Zakho 18
Growth Rates
Constant 1
Logarithmic logn
Linear n
Quadratic n2
Cubic n3
Polynomial nk (for k ≥ 1)
Exponential an (a ≥ 1)
Growth rate is not affected by
– constant factors or
– lower-order terms
Ex: 102n + 105 is a linear function
Ex: 105n2 + 108n is a quadratic function
Data Structures Department of Computer Science – University of Zakho 19
Asymptotic Complexity
• Worst case running time of an algorithm as a function of input size n
for large n.
• Expressed using only the highest-order term in the expression for the
exact running time.
– Instead of exact running time, say O(n2)
• Written using asymptotic notation (O, , , o, )
– Ex: f(n) = O(n2)
– Describes how f(n) grows in comparison to n2
• The notations describe different rate-of-growth relations between
the defining function and the defined set of functions
Data Structures Department of Computer Science – University of Zakho 20
O-notation
For functions g(n), we define
O(g(n)), big-O of n, as the set:
O(g(n)) = { f(n) :
∃ positive constants c and n0,
such that ∀n ≥ n0
we have 0 f(n) cg(n) }
Intuitively: Set of all functions whose rate of growth is the Technically, f(n) ∈ O(g(n)).
same as or lower than that of g(n). Older usage, f(n) = O(g(n)).
g(n) is an asymptotic upper bound for f(n)
Data Structures Department of Computer Science – University of Zakho 21
Examples
O(g(n)) = { f(n) : ∃ positive constants c and n0,
such that ∀n ≥ n0 , we have 0 f(n) cg(n) }
• O(n) O(n2)
f(n)=7n+3 f(n) = n2 + 1
f(n) = 2n + 10 f(n) = n2 + n
f(n) = n + 1 f(n) = 10000n2 +10000n + 300
f(n) = 10000n f(n) = n1.99
f(n) = 10000n + 300
• The function n2 is not O(n)
– the inequality n2 cn cannot be satisfied since c is constant
Data Structures Department of Computer Science – University of Zakho 22
Big-Oh Rules
Drop lower-order terms
Ex: if f(n) is a polynomial of degree d, then f(n) is O(nd)
Drop constant factors, using the simplest expression of the class
Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
Use the smallest possible class of functions
Say “2n is O(n)” instead of “2n is O(2n)”
Data Structures Department of Computer Science – University of Zakho 23
Asymptotic Algorithm Analysis
The asymptotic analysis of an algorithm determines the running time
in big-Oh notation
To perform the asymptotic analysis
Find the worst-case number of primitive operations executed as a function of
the input size
We express this function with big-Oh notation
Ex:
arrayMax executes at most 7n 1 primitive operations
arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually
dropped anyhow, we can disregard them when counting primitive
operations
Data Structures Department of Computer Science – University of Zakho 24
Ex: Computing Prefix Averages
We further illustrate asymptotic analysis
with two algorithms for prefix averages. 35
X
30 A
The i-th prefix average of an array X is
average of the first (i + 1) elements of X:
25
A[i] (X[0] + X[1] + … + X[i])/(i+1) 20
15
Prefix average has applications in 10
economic and statistics
5
0
1 2 3 4 5 6 7
Data Structures Department of Computer Science – University of Zakho 25
Prefix Averages V1 O(n2) - Quadratic!
The following algorithm computes prefix averages by
applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X rough # operations
A new array of n integers n
for i 0 to n 1 do n
s X[0] n
for j 1 to i do 1 + 2 + … + (n-1)
s s + X[j] 1 + 2 + … + (n-1)
A[i] s / (i + 1) n
return A 1
Analysis of Algorithms 26
Data Structures Department of Computer Science – University of Zakho 26
Prefix Averages V2
O(n) - Linear!
The following algorithm computes prefix averages
by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
rough # operations
Output array A of prefix averages of X
A new array of n integers n
s0 1
for i 0 to n 1 do n
s s + X[i] n
A[i] s / (i + 1) n
return A 1
Analysis of Algorithms 27
Data Structures Department of Computer Science – University of Zakho 27
Ω-notation
For functions g(n), we define
Ω(g(n)), big-Omega of n, as the
set:
Ω(g(n)) = { f(n) :
∃ positive constants c and n0,
such that ∀n ≥ n0
we have 0 cg(n) f(n)}
Intuitively: Set of all functions whose rate of growth is the
same as or higher than that of g(n).
g(n) is an asymptotic lower bound for f(n)
Analysis of Algorithms 28
Data Structures Department of Computer Science – University of Zakho 28
𝛩-notation
For functions g(n), we define
𝛩(g(n)), big-Theta of n, as the set:
𝛩(g(n)) = { f(n) :
∃ positive constants c1, c2, and n0,
such that ∀n ≥ n0
we have 0 c1g(n) f(n) c2g(n)}
Intuitively: Set of all functions that have the same rate of
growth as g(n).
g(n) is an asymptotically tight bound for
f(n)
Analysis of Algorithms 29
Data Structures Department of Computer Science – University of Zakho 29
Relationship between O, Ω, 𝛩
Analysis of Algorithms 30
Data Structures Department of Computer Science – University of Zakho 30
