DESIGN AND ANALYSIS

OF AN ALGORITHM
ALGORITHM COMPLEXITY

Ngo Quoc Viet-2023

Contents
• Algorithm Complexity
• How to determine T(n)
• Complexity of sequential, conditional commands
• Complexity of loop, nested commands
• Big-O, big-Omega, and big-Theta.
• Complexity of recursive algorithm
Lecture outcomes
• Understanding roles of algorithm complexity in academic and
practice.
• Know how to determine complexity of the simple algorithms via
counting techniques.
• Know how to determine and prove growth rate of function, via
bigO.
Analysis algorithm complexity
• The goal of algorithm analysis is to take a block of code and determine the
asymptotic run time or asymptotic memory requirements based on various
parameters
• Complexity of an algorithm is a measure of the amount of time and/or space
required by an algorithm for an input of a given size n.
• For example: sorting the array of size n.
• Selection sort: ~𝑛2 time complexity.
• Merge sort, quick sort, heap sort: ~𝑛𝑙𝑜𝑔𝑛 time complexity.

• Merge sort requires ~𝑛 additional memory

• Quick sort requires ~𝑙𝑜𝑔𝑛 additional memory
• Heap sort requires ~1 memory
Analysis algorithm complexity
• The asymptotes represent the scale of the algorithm. Given an array of
size n.
• Time complexity needs
• Selection sort: requires Θ 𝑛2 ⇒ 2𝑛 entries require 4𝑛2 (4 times), 10𝑛 entries
require 100𝑛2 (100 times).
• Merge/Quick/Heap sort: requires Θ(𝑛𝑙𝑜𝑔𝑛) ⇒ 2𝑛 entries require 2𝑛𝑙𝑜𝑔(2𝑛) =
2(𝑛𝑙𝑜𝑔𝑛) + 2𝑛, 10𝑛 entries need 10𝑛𝑙𝑜𝑔(10𝑛) = 10(𝑛𝑙𝑜𝑔𝑛) + 10𝑛.
• Memory needs
• Merge sort requires twice and 10 times as much memory.
• Heap sort doesn't need extra memory.
Analysis algorithm complexity
• Eg: Time of running algorithm of input size of n is Θ 𝑛𝑙𝑜𝑔3 , log 3 ≈
1.585. Double n, It requires (2𝑛)𝑙𝑜𝑔3 = 2𝑙𝑜𝑔3 . 𝑛𝑙𝑜𝑔3 = 3 𝑛𝑙𝑜𝑔3 ; If 10
times of input size, It requires (10𝑛)𝑙𝑜𝑔3 = 10𝑙𝑜𝑔3 . 𝑛𝑙𝑜𝑔3 = 38.5 𝑛𝑙𝑜𝑔3
(nearly 38 times slower).
• If a hash table is used for storage, then in many cases functions will
have optimal runtime.
• Assume that each instruction takes Θ 1 execution time, regardless
of device hardware.
Analysis algorithm complexity
• We need to determine the a function T(n) that represents the
complexity. The function T(n) is called the complexity of the
algorithm.
• It should be to indicate the worst case, average, and best cases of T(n).
• Other factors such as: hardware, programming language, etc. are
ignored when considering complexity.
• In practice, the complexity is expressed through the growth of the
function with big-O, big-Omega, big-Theta based on the simple
formulas.
Some popular formulas in algorithm complexity

n logn nlogn n2 n3 2n
1 0 0 1 1 2
2 1 2 4 8 4
3 2 8 16 64 16
8 3 24 64 512 256
16 4 64 256 4096 65536
32 5 160 1024 32768 2147483648
The techniques to determine T(n)
• Computing/Counting statement by statement
• Sequential, conditional, loop, nested loop statements.
• https://www.youtube.com/watch?v=8syQKTdgdzc
• Probability theory to determine average complexity.
• Mathematical methods in the analysis of algorithms -combinatorial
enumeration problems
(https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.77.2198&r
ep=rep1&type=pdf)
• For function growth rate, we can use the sum and multiply rules for big-
O, big-Theta, big-Omega.
Conditional command analysis
if ( condition ) {
// true body
} else {
// false body
}

• Runtime of conditional command:

• Run time of the if condition, plus
• The body of the block of statements executes according to the condition
that is met.
• Normally, the run time of the if condition is Θ 1 .
Conditional command analysis
• In many cases, determining the runtime of a command block is not easy, for example
int find_max( int *array, int n ) {
max = array[0]; C1: 1
for (int i = 1; i < n; i+=1) { C2: (n-1+1) // upper-lower+1. n-1+1
if ( array[i] > max) { C3: (n-1) // upper-lower
max = array[i]; C4: (n-1) // worst – ascent. # Times = n-1
// best – maximum is in [0]. # Times = 0
// avg = (n-1+0)/2
}
}
return max; C5: 1
}
T(n)worst=1+n+(n-1)+(n-1)+1=3n-1; Tbest(n) = 1+n+(n-1)+0+1=2n+1; Tavg(n)=1+n+(n-1+0)/2+1=(3/2)n+3.5
• It depends on the array increasing, decreasing or unordered
Loop command analysis
for ( int i = 0; i < N; i ++ ) { (N + 1)
// code which is Theta(f(N)) N
T(N) = N + 1 + N = 2N + 1
}
Run time of initialization, condition Θ 1

• If there is no break or return in the loop, the run time is Θ 𝑁. 𝑓 𝑁 .

int sum = 0; C1: 1
for (int i = 0; i < n; i += 1) { C2: (n-0+1) 𝑛−1
𝑛+1 𝑛+2
෍ (𝑛 − 𝑖 + 1) = −1
2
for (int j = i; j < n; ++j ) { C3: (n-i)+1 𝑖=0

𝑛−1
sum += 1; Theta(1) C4: (n-i)
෍ (𝑛 − 𝑖) = 𝑛 + 1 𝑛/2
} 𝑖=0

} T(n)=C1+C2+C3+C4
Loop command analysis
void Disjoint_sets::clear() {
if ( sets == n ) { Θ 1
return;
}

Θ 1 , 𝑠𝑒𝑡𝑠 = 𝑛
max_height = 0; 𝑇𝑐𝑙𝑒𝑎𝑟 𝑛 = ቊ
Θ 𝑛 , 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
num_disjoint_sets = n; Θ 1

for ( int i = 0; i < n; ++i ) { n+1: Θ 𝑛

parent[i] = i; n
tree_height[i] = 0; n
}
}
Loop command analysis
• If the loop body depends on a counter variable (variable i in the
following example), then the run time of
for ( int i = 0; i < n; ++i ) { n-0+1
// code which is Theta(f(i,n)) n*f(i,n)
}
𝑛−1

1 + ෍ 1 + 𝑓 𝑖, 𝑛
𝑖=0
Loop command analysis
int sum = 0; C1: 1
for ( int i = 0; i < n; ++i ) { C2: (n+1)
for ( int j = 0; j < i; ++j) {C3: sigma(i+1, 0, n-1)
sum += i + j; C4: sigma(i, 0, n-1)
sum -= 1; C5: sigma(i, 0, n-1)
}
}
• The inner loop is Θ 1 + 𝑖(1 + 1) = Θ 𝑖 , outer loop is
𝑛−1 𝑛−1
𝑛(𝑛 − 1)
Θ 1+෍ 1+𝑖 =Θ 1+𝑛+෍𝑖 = Θ 1+𝑛+ = Θ 𝑛2
2
𝑖=0 𝑖=0
Examples of algorithm complexity
• Multiply matrix
void matrixmult(int n, A[][], B[][], C[][]) {
for (i = 0; i < n; i++) C1: n+1 <= 2*n O(n)
for (j = 0; j < i; j++){ C2: i+1 <= 1*n O(n)
C[i][j] = 0; C3: 1 O(1)
for (k = 0; k < j; k++) C4: j+1 <= 1*n O(n)
C[i][j]=C[i][j]+A[i][k]*B[k][j]; O(1)
}
}
O(n*n*1*n*1) = O(n^3)
Examples of algorithm complexity
• Insert sort
for(i=2; i <= n; i ++) { C1: n
x = a[i]; j = i - 1;
while( (j > 0) && a[j] > x) {
a[j+1]=a[j];
j --;
}
a[j+1]=x; C6: (n-1)
}
Examples of algorithm complexity
• Best case: 𝐵𝑐𝑜𝑚𝑝 𝑛 = 𝑛 − 1
• Worst case:
𝑛(𝑛−1)
𝑊𝑐𝑜𝑚𝑝 𝑛 = σ𝑛𝑖=2 𝑖−1 = = 𝑂 𝑛2
2
• Average case: ith iteration, at most i positions to insert x → the
probability to insert x at a specified position in step i is 1/𝑖.
Examples of algorithm complexity
• Number of comparisons if insert x in step i at position Location (j Number of comparisons
value) x < a[i]
1 i
• Number of iterations in step i.
2 i-1
3 i-2
𝑖−1 ...
1 1 1 1 1𝑖 𝑖 − 1 𝑖−1
1. + 2. + ⋯ + 𝑖 − 1 = ෍ 𝑘 = = i-2 2
𝑖 𝑖 𝑖 𝑖 𝑖 2 2 i-1 1
𝑘=1

• Number of iterations for the whole array:

𝑛
𝑖−1
෍
2
𝑖=2
Examples of algorithm complexity
• Closest pair of points
Examples of algorithm complexity
• Given a graph of n nodes, list every k node subgraph such that the
two nodes are not joined by an edge
Big-O, Big-Omega, Big-Theta
• Definition: Given a function T(n), T(n) is said to have complexity f(n) if
there exist constants C, n0 such that 𝑇 𝑛 ≤ 𝐶𝑓 𝑛 , ∀𝑛 ≥ 𝑛0 . T(n) has an
growth rate of f(n)-and the symbol is 𝑂(𝑓(𝑛)) (the big-O of f(n)).
• Eg: 𝑇(𝑛) = (𝑛 + 1)2 has an growth rate of 𝑛2, 𝑇(𝑛) thuộc 𝑂(𝑛2).
• For any constant positive integer K
𝑂(𝐾. 𝑓(𝑛)) = 𝑂(𝑓(𝑛))
1
𝑂(𝑓( 𝑛)) = 𝑂(𝑓(𝑛))
2
𝑂(𝐾) = 𝑂(1)
Big-O, Big-Omega, Big-Theta
• The idea of big-O is to find the upper bound of the growth of the
function 𝑇(𝑛) when n is large enough.
• The upper bound is determined by the function 𝑓(𝑛) , simpler than
𝑇(𝑛).
• Determine the constant C such that 𝑇 𝑛 ≤ 𝐶𝑓 𝑛 , ∀𝑛 ≥ 𝑛0
• The goal is to find the smallest upper bound 𝑓(𝑛), such that 𝑇 𝑛 ≤
𝐶𝑓 𝑛 , ∀𝑛 ≥ 𝑛0 .
Big-O, Big-Omega, Big-Theta
• Theorem: Given Polynomial
𝑑

𝑇(𝑛) = ෍ 𝑎𝑖 𝑛𝑖
𝑖=0
Then
𝑇(𝑛) ∈ 𝑂(𝑛𝑑 )
• Eg: Given 𝑇 𝑛 = 𝑛2 + 2𝑛 + 1, It is easy to
see 𝑛2 + 2𝑛 + 1 ≤ 2𝑛2 . 𝑇 𝑛 ≤ 2. 𝑛2 , ∀𝑛 ≥ 3
⇒ 𝑇(𝑛) ∈ 𝑂 𝑛2
Some properties of big-O

Rule

Addition rule: Multiplication rule:

The execution time of two The execution time of
consecutive blocks is the two nested program
segments is:
𝑻(𝒏) = 𝑶(max(𝒇(𝒏), 𝒈(𝒏))) 𝑻(𝒏) = 𝑶(𝒇(𝒏). 𝒈(𝒏))
Big-O, Big-Omega, Big-Theta
Big-O, Big-Omega, Big-Theta
• Two concepts that are similar to big-O are: big- and big-.
• Definition of big-: Given the function g(n), then Ω(𝑔(𝑛)) is a set of
functions.
Ω(𝑔(𝑛)) = {𝑇(𝑛): 𝑐 𝑎𝑛𝑑 𝑛0 , 0 ≤ 𝑐𝑔(𝑛) ≤ 𝑇(𝑛), ∀𝑛 ≥ 𝑛0}
• g(n) is the lower asymptote of T(n).
𝑇(𝑛) ∈ Ω(𝑔(𝑛))
• big- represents minimal complexity.
Big-O, Big-Omega, Big-Theta
• Big-: upper and lower bounds
𝑓 𝑥 = Θ 𝑔 𝑥 ⟺𝑓 𝑥 =𝑂 𝑔 𝑥 ∧ 𝑓 𝑥 =Ω 𝑔 𝑥
• Comment: every polynomial is a big- of the largest exponential.
𝑥 4 /100000 + 3𝑥 3 + 5𝑥 2 – 9 ∈ Θ(𝑥 4 )
Big-O, Big-Omega, Big-Theta
• Sort the following functions from smallest to largest
1
𝑥 + 𝑠𝑖𝑛𝑥; ln 𝑥 ; 𝑥 + 𝑥; ;
𝑥
13 + 𝑥; 𝑒 𝑥 ; 𝑥 𝑒 ; 𝑥 𝑥 ;
𝑥 + 𝑠𝑖𝑛𝑥 𝑥 20 − 101 ; 𝑥𝑙𝑛 𝑥; 𝑥 ln 𝑥 2 ; 𝑙𝑔2 𝑥

1 1
; 13 + ; ln 𝑥 ; 𝑙𝑔2 𝑥; 13 + 𝑥; 𝑥 + 𝑠𝑖𝑛𝑥; 𝑥 + 𝑥; 𝑥𝑙𝑛 𝑥; 𝑥 ln 𝑥 2 ; 𝑥 𝑒 ;
𝑥 𝑥
𝑥 + 𝑠𝑖𝑛𝑥 𝑥 20 − 101 ; 𝑒 𝑥 ; 𝑥 𝑥
Recursive function analysis
int factorial( int n ) {
if ( n <= 1 ) {
return 1;
} else {
return n * factorial( n – 1 ); 𝑇 𝑛 =𝑇 𝑛−1 +1
}
}
Recursive function analysis
• Complexity of the factorial function
𝑂(1) 𝑛≤1
𝑇( 𝑛) = ቊ
𝑇( 𝑛 − 1) + 𝑂(1) 𝑛>1
• Using MAPLE:
rsolve({T(n) = T(n – 1) + 1, T(1) = 1}, T(n) );
𝑇( 𝑛) = 𝑂 𝑛
Recursive function analysis
𝑇(𝑛) = 𝑇(𝑛 – 1) + 1
= 𝑇(𝑛 – 2) + 1 + 1 = 𝑇(𝑛 – 2) + 2
= 𝑇(𝑛 – 3) + 3
• From this, we see a pattern:
𝑇(𝑛) = 𝑇(𝑛 – 𝑘) + 𝑘
If k = n – 1 then
T(n) = T(n – (n – 1)) + n – 1
= T(1) + n – 1
=1+n–1=n

𝑇(𝑛) = 𝑂(𝑛)
Recursive function analysis
void sort( int * array, int n ) {

if ( n <= 1 )

return; // special case: 0 or 1 items are always sorted

int posn = 0; // assume the first entry is the smallest

int max = array[posn];

for ( int i = 1; i < n; ++i ) // search through the remaining entries

if ( array[i] > max ) { // if a larger one is found

posn = i; // update both the position and value

max = array[posn];

int tmp = array[n - 1]; // swap the largest entry with the last

array[n - 1] = array[posn]; array[posn] = tmp;

sort( array, n – 1 ); // sort everything else

}
Recursive function analysis
Recursive function analysis
n 
• rsolve( {T(n) = T(n – 1) + n, T(1) = 1}, T(n) ); −1 − n + ( n + 1 )  + 1 
2 
1 1 2
• expand( % ); n+ n
2 2
• Manually solve the complexity of Selection Sort
T( 𝑛) = T( 𝑛 − 1) + 𝑛
= T( 𝑛 − 2) + (𝑛 − 1) + 𝑛
= T( 𝑛 − 2) + 𝑛 + (𝑛 − 1)
= T( 𝑛 − 3) + 𝑛 + (𝑛 − 1) + (𝑛 − 2)
⋮
𝑛 𝑛 𝑛

= T( 1) + ෍ 𝑖 = 1 + ෍ 𝑖 = ෍ 𝑖
𝑖=2 𝑖=2 𝑖=1
Recursive binary search analysis
• Check the middle element, if not found check the left range or right range
T( 𝑛) = T((n – 1)/2)+Θ 1
• Solve
1, 𝑛=1
T( 𝑛) = 𝑓 𝑥 = ൞ 𝑛 − 1
𝑇 + 1, 𝑛>1
2
𝑛−1
𝑛= 2𝑘 −1⟹ = 2𝑘−1 − 1
2
Recursive binary search analysis
T( 𝑛) = T( 2𝑘 − 1)
2𝑘 − 1 − 1
=T +1
2
= T( 2𝑘−1 − 1) + 1
2𝑘−1 − 1 − 1
=T +1+1
2
= T( 2𝑘−2 − 1) + 2
⋮
T( 𝑛) = T( 2𝑘 − 1)
= T( 2𝑘−(𝑘−1) − 1) + 𝑘 − 1
= T( 1) + 𝑘 − 1 = 𝑘
 n
Master theorem
n-1
• Given the function n-2
Function T( n : size) :
if n < 1 exit
Do work of amount f(n) n-2 n-3 n-3 n-4
T(n/b)
T(n/b)
...repeat for a total of a times...
T(n/b)
End
𝑛
• Then: 𝑇 𝑛 = 𝑎𝑇 + 𝑓(𝑛), f(n) is the implementation cost outside
𝑏
of recursion.
Master theorem
• Theorem: suppose 𝑓(𝑛) = 𝑂 𝑛𝑐
• Case 1: If 𝑐 < 𝑙𝑜𝑔𝑏 𝑎 then 𝑇 𝑛 = Θ 𝑛𝑙𝑜𝑔𝑏 𝑎
• Case 2: If 𝑐 = 𝑙𝑜𝑔𝑏 𝑎 then 𝑇 𝑛 = Θ 𝑛𝑐 𝑙𝑜𝑔𝑛
• Case 3: If 𝑐 > 𝑙𝑜𝑔𝑏 𝑎 then 𝑇 𝑛 = Θ 𝑛𝑐
• Example
𝑇 𝑛 = 8𝑇 𝑛/2 + 1000𝑛2 ( 𝑐 = 2 < 𝑙𝑜𝑔𝑏 𝑎 = 𝑙𝑜𝑔2 8 = 3 → 𝑇 𝑛 =
Θ 𝑛3
𝑇 𝑛 = 2𝑇 𝑛/2 + 10𝑛 ( 𝑐 = 1 = 𝑙𝑜𝑔𝑏 𝑎 = 𝑙𝑜𝑔2 2 = 1 → 𝑇 𝑛 =
Θ 𝑛𝑙𝑜𝑔𝑛
𝑇 𝑛 = 2𝑇 𝑛/2 + 𝑛2 ( 𝑐 = 2 > 𝑙𝑜𝑔𝑏 𝑎 = 𝑙𝑜𝑔2 2 = 1 → 𝑇 𝑛 = Θ 𝑛2
Master theorem
• Case 1: if 𝑓 𝑛 = 𝑂 𝑛𝑙𝑜𝑔𝑏𝑎−𝜀 , 𝜀 > 0, then 𝑇 𝑛 = Θ 𝑛𝑙𝑜𝑔𝑏 𝑎 .

• Case 2: if 𝑓(𝑛) = Θ 𝑛𝑙𝑜𝑔𝑏𝑎 𝑙𝑜𝑔𝑘 𝑛 then 𝑇 𝑛 = Θ 𝑛𝑙𝑜𝑔𝑏𝑎 𝑙𝑜𝑔𝑘+1 𝑛 .

𝑛
• Case 3: if 𝑓(𝑛) = Ω 𝑛𝑙𝑜𝑔𝑏 𝑎+𝜀 , and a𝑓 ≤ 𝑐𝑓 𝑛 , 𝑐 < 1 , then
𝑏
𝑇 𝑛 = Θ 𝑓(𝑛) .
Master theorem
• Determine the complexity of the following
expressions, or determine that master's
theorem cannot be applied.
• 𝑇 𝑛 = 3𝑇 𝑛/2 + 𝑛2 – Θ 𝑛2 (case 3)
• 𝑇 𝑛 = 4𝑇 𝑛/2 + 𝑛2 – Θ 𝑛2 𝑙𝑜𝑔𝑛 (case 2))
• 𝑇 𝑛 = 𝑇 𝑛/2 + 2𝑛 – Θ 2𝑛 (case 3)
• 𝑇 𝑛 = 2𝑛 𝑇 𝑛/2 + 𝑛𝑛 – 𝑁𝑜𝑡 (a: not const)
• 𝑇 𝑛 = 16𝑇 𝑛/4 + 𝑛 – Θ 𝑛2 (case 1)
– 𝑛 𝑙𝑜𝑔2 𝑛
• 𝑇 𝑛 = 2𝑇 𝑛/2 + 𝑛𝑙𝑜𝑔𝑛
– 𝑁𝑜𝑡 (non-polynomial difference
• 𝑇 𝑛 = 2𝑇 𝑛/2 + 𝑛/𝑙𝑜𝑔𝑛 𝑎
between f(n) and 𝑛 𝑙𝑜𝑔𝑏 )
Master theorem
− 𝑇 𝑛 = 2𝑇 𝑛/4 + 𝑛0.51 − Θ 𝑛0.51 (case 3)
1 − 𝑁𝑜𝑡 (a < 1)
− 𝑇 𝑛 = 0.5𝑇 𝑛/2 +
𝑛
− Θ 𝑛! (case 3)
− 𝑇 𝑛 = 16𝑇 𝑛/4 + 𝑛!
− Θ 𝑛 (case 1)
− 𝑇 𝑛 = 2𝑇 𝑛/2 + 𝑙𝑜𝑔𝑛
− Θ 𝑛𝑙𝑜𝑔3 (case 1)
− 𝑇 𝑛 = 3𝑇 𝑛/2 + 𝑛
− Θ 𝑛 (case 1)
− 𝑇 𝑛 = 3𝑇 𝑛/3 + 𝑛 − Θ 𝑛2 (case 1)
− 𝑇 𝑛 = 4𝑇 𝑛/2 + 𝑐𝑛 − Θ 𝑛𝑙𝑜𝑔𝑛 (case 3)
− 𝑇 𝑛 = 3𝑇 𝑛/4 + 𝑛𝑙𝑜𝑔𝑛 − Θ 𝑛𝑙𝑜𝑔𝑛 (case 3)
− 𝑇 𝑛 = 3𝑇 𝑛/3 + 𝑛/2 − Θ 𝑛2 𝑙𝑜𝑔𝑛 (case 2)
− 𝑇 𝑛 = 6𝑇 𝑛/3 + 𝑛2 𝑙𝑜𝑔𝑛 − Θ 𝑛2 (case 1)
− 𝑇 𝑛 = 4𝑇 𝑛/2 + 𝑙𝑜𝑔𝑛/𝑛
Summary
• Using counting method to determine algorithm complexity
• Solve the regression function to determine recursive algorithm
complexity.
• Big-O, Big-Omega, Big-Theta are used to represent algorithm
complexity.
• Master theorem can be used to determine big-O of the recursive
algorithms.