TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI Course outline

VIỆN CÔNG NGHỆ THÔNG TIN VÀ TRUYỀN THÔNG

Chapter 1. Fundamentals
Chapter 2. Algorithmic paradigms
Chapter 3. Basic data structures
Data structures and Algorithms Chapter 4. Tree
Chapter 5. Sorting
Nguyễn Khánh Phương

Computer Science department

Chapter 6. Searching
School of Information and Communication technology
E-mail: [email protected] Chapter 7. Graph

TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI

VIỆN CÔNG NGHỆ THÔNG TIN VÀ TRUYỀN THÔNG
Contents
1.1. Introductory Example
1.2. Algorithm and Complexity
1.3. Asymptotic notation
1.4. Running time calculation
Chapter 1. Fundamentals

Nguyễn Khánh Phương

Computer Science department

School of Information and Communication technology
E-mail: [email protected]

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

1
Contents Example: The maximum subarray problem
1.1. Introductory Example • Given an array of n numbers:

a1, a2, … , an
1.2. Algorithm and Complexity
The contiguous subarray ai, ai+1 , …, aj with 1 ≤ i ≤ j ≤ n is a subarray of
1.3. Asymptotic notation the given array and ∑ 𝑎 is called as the value of this subarray
1.4. Running time calculation The task is to find the maximum value of all possible subarrays, in other
words, find the maximum ∑ 𝑎 . The subarray with the maximum value is
called as the maximum subarray.
Example: Given the array -2, 11, -4, 13, -5, 2 then the maximum subarray is
11, -4, 13 with the value = 11+ (-4)+13 =20

 This problem can be solved using several different algorithmic techniques,

NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST
including brute force, divide and conquer, dynamic programming, etc.

1. Introductory example: the max subarray problem 1. Introductory example: the max subarray problem

1.1.1. Brute force 1.1.1. Brute force

1.1.2. Brute force with better implement 1.1.2. Brute force with better implement
1.1.3. Dynamic programming 1.1.3. Dynamic programming

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

2
1.1.1. Brute force algorithm to solve max subarray problem Brute force algorithm: browse all possible sub-array

• The first simple algorithm that one could think about is: Index i 0 1 2 3 4 5
a[i] -2 11 -4 13 -5 2
browse all possible sub-arrays:
i = 0: (-2), (-2, 11), (-2,11, -4), (-2,11,-4,13), (-2,11,-
4,13,-5), (-2,11,-4,13,-5,2)
ai, ai+1 , …, aj với 1 ≤ i ≤ j ≤ n,
i = 1: (11), (11, -4), (11, -4, 13), (11, -4, 13, -5), (11, -4,
13, -5, 2)
then calculate the value of each sub-array in order to find i = 2: (-4), (-4, 13), (-4, 13, -5), (-4,13,-5,2)
i = 3: (13), (13,-5), (13, -5,2)
the maximum value. i = 4: (-5), (-5, 2) int maxSum = a[0];
i = 5: (2) for (int i=0; i<n; i++) {

• The number of all possible sub-arrays: for (int j=i; j<n; j++) {
int sum = 0;
for (int k=i; k<=j; k++)
C(n, 1) + C(n, 2) = n2/2 + n/2 sum += a[k];
if (sum > maxSum)
maxSum = sum;
NGUYỄN KHÁNH PHƯƠNG }
SOICT– HUST
}

Brute force algorithm: browse all possible sub-array 1. Introductory example: the max subarray problem
• Analyzing time complexity: we count the number of additions that the algorithm need to
perform, it means we count the statement
1.1.1. Brute force
sum += a[k] 1.1.2. Brute force with better implement
must perform how many times.
The number of additions: 1.1.3. Dynamic programming
n 1 n 1 n 1 n 1
(n  i )(n  i  1)
 ( j  i  1)  (1  2  ...  (n  i))  
i  0 j i i 0 i 0 2
1 n 1  n 2 n  1  n( n  1)(2n  1) n(n  1) 
  k (k  1)  2 
2 k 1 k 1
k  k  
k 1  2 6

2 
3
n n 2
n int maxSum = a[0];
   for (int i=0; i<n; i++) {
6 2 3
for (int j=i; j<n; j++) {
int sum = 0;
for (int k=i; k<=j; k++)
sum += a[k];
if (sum > maxSum)
maxSum = sum;
} NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST
}

3
1.1.2. A better implementation 1.1.2. A better implementation
Brute force algorithm: browse all possible sub-array Brute force algorithm: browse all possible sub-array
Index i 0 1 2 3 4 5 Index i 0 1 2 3 4 5
a[i] -2 11 -4 13 -5 2 a[i] -2 11 -4 13 -5 2
i = 0: 18 + (-5)=13 13
i = 0: (-2), (-2, 11), (-2,11, -4), (-2,11,-4,13), (-2,11,-4,13,-5), (-2,11,-
9 + (-4)=5 5 4,13,-5,2)
i = 1: (11), (11, -4), (11, -4, 13), (11, -4, 13, -5), (11, -4, 13, -5, 2)
(-2),(-2, 11), (-2,11, -4), (-2,11,-4,13), (-2,11,-4,13,-5), (-2,11,-4,13,-5,2)
i = 2: (-4), (-4, 13), (-4, 13, -5), (-4,13,-5,2)
-2+11 = 9 9 5+13=18 18 i = 3: (13), (13,-5), (13, -5,2)
i = 4: (-5), (-5, 2)
i = 5: (2)

int maxSum = a[0];//or maxSum=-∞; int maxSum = a[0];

for (int i=0; i<n; i++) { for (int i=0; i<n; i++) {
for (int j=i; j<n; j++) { int sum = 0;
int sum = 0; for (int j=i; j<n; j++) {
for (int k=i; k<=j; k++) sum += a[j];
sum += a[k]; if (sum > maxSum)
if (sum > maxSum) maxSum = sum;
maxSum = sum; }
NGUYỄN KHÁNH PHƯƠNG } }
SOICT– HUST 13 } 14

1.1.2. A better implementation 1.1.2. A better implementation

Brute force algorithm: browse all possible sub-array Brute force algorithm: browse all possible sub-array
• A better implementation: • Analyzing time complexity: we again count the number of additions that the
We could get the sum of elements from i to j by just using one addition: algorithm need to perform, it means we count the statement
j j 1 Sum += a[j]

 a[k ]  a[ j ]   a[k ] must perform how many times.

k i k i The number of additions:
n 1
n2 n
The sum of elements from i to j The sum of elements from i to j-1  (n  i)  n  (n  1)  ...  1 
i 0

2 2
This number is exactly the number of all possible sub-arrays  it seems this
int maxSum = a[0]; int maxSum = a[0]; implementation is good as we examine each subarray exactly once.
for (int i=0; i<n; i++) { for (int i=0; i<n; i++) {
for (int j=i; j<n; j++) { int maxSum = a[0];
int sum = 0;
int sum = 0; for (int i=0; i<n; i++) {
for (int j=i; j<n; j++) {
for (int k=i; k<=j; k++) int sum = 0;
sum += a[j];
sum += a[k]; for (int j=i; j<n; j++) {
if (sum > maxSum)
if (sum > maxSum) sum += a[j];
maxSum = sum;
maxSum = sum; if (sum > maxSum)
}
} maxSum = sum;
}
} }
}

4
Max subarray problem: compare the time complexity Max subarray problem: compare the time complexity between
between algorithms algorithms
The number of additions that the algorithm need to perform: Complexity n=10 Time (sec) n=100 Time (sec) n=104 Time n=106 Time

1.1.1. Brute force n3 103 10-5 106 10-2 sec 1012 2.7 hours 1018 115 days
n2 100 10-6 10000 10-4 sec 108 1 sec 1012 2.7 hours

1.1.2. Brute force with better implement

 For the same problem (max subarray), we propose 2 algorithms that requires • With small n, the calculation time is negligible.
different number of addition operations, and therefore, they will require different • The problem becomes more serious when n > 106. At that time, only
computation time. the third algorithm is applicable in real time.
The following tables show the computation time of these 2 algorithms with the • Can we do better?
assumption: the computer could do 108 addition operation per second Yes! It is possible to propose an algorithm that requires only n additions!
Complexity n=10 Time (sec) n=100 Time (sec) n=104 Time n=106 Time
n3 103 10-5 106 10-2 sec 1012 2.7 hours 1018 115 days
n2 100 10-6 10000 10-4 sec 108 1 sec 1012 2.7 hours

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

1. Introductory example: the max subarray problem 1.1.3. Dynamic programming to solve max subarray problem
1.1.1. Brute force The primary steps of dynamic programming:
1. Divide: Partition the given problem into sub problems

(Sub problem: have the same structure as the given problem but with smaller size)
1.1.2. Brute force with better implement
2. Note the solution: store the solutions of sub problems in a table

3. Construct the final solution: from the solutions of smaller size problems, try to
1.1.3. Dynamic programming find the way to construct the solutions of the larger size problems until get the
solution of the given problem (the sub problem with largest size)
n

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

5
1.1.3. Dynamic programming to solve max subarray problem 1.1.3. Dynamic programming to solve max subarray problem
The primary steps of dynamic programming: MaxSub(a)
{
1. Divide: smax = a[0]; // smax : the value of max subarray
• Define si the value of max subarray of the array a0, a1, ..., ai , i = 0, 1, 2, ..., n-1. ei = a[0]; // ei : the value of max subarray ending at a[i]
imax = 0; // imax : the index of the last element of the max sub array
• Clearly, sn-1 is the solution.
for i = 1 to n-1 {
u = ei + a[i];
3. Construct the final solution:
v = a[i];
• s0 = a0 s1 = max{a0, a1, a0 + a1} if (u > v) ei = u;
• Assume we already know the value of s0, s1, s2,…, si-1, i >=1. Now we need to calculate the value of si which is else ei = v;
the value of max subarray of the array: if (ei > smax) {
smax = ei;
a0, a1, ..., ai-1, ai .
imax = i;
• We see that: the max subarray of this array a0, a1, ..., ai-1, ai could either include the element ai or not include }
the element ai  therefore, the max subarray of the array a0, a1, ..., ai-1, ai could only be one of these 2 arrays: }
}
– The max subarray of the array a0, a1, ..., ai-1 = si-1
– The max subarray of the array a0, a1, ..., ai ending at ai. = ei Analyzing time complexity:
 Thus, we have si = max {si-1, ei}, i = 1,2, …, n-1. the number of addition operations need to be performed in the algorithm
where ei is the value of the max subarray a0, a1, ..., ai ending at ai. = the number of times the statement u = ei + a[i]; need to be executed
To calculate ei, we could use the recursive relation: =n
– e0 = a0;
– ei = max {ai, ei-1 + ai}, i = 1,2, ..., n-1. NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

Comparison of 3 algorithms Contents

• The following table shows the estimated running time of the four proposed 1.1. Introductory Example
algorithms above (assuming the computer could perform 108 addition
operations per second).
1.2. Algorithm and Complexity
1.3. Asymptotic notation
Algorithm Complexity n=104 time n=106 time
Brute force n3 1012 2.7 hours 1018 115 days 1.4. Running time calculation
Brute force n2 108 1 sec 1012 2.7 hours
with better
implementation
Dynamic n 104 10-4 sec 106 2*10-2 sec
programming

This example shows how the development of effective algorithms could

significantly reduce the cost of running time.
NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

6
Algorithm Algorithm
• The word algorithm comes from the name of a Persian mathematician Abu Ja’far • All algorithms must satisfy the following criteria:
Mohammed ibn-i Musa al Khowarizmi.
• In computer science, this word refers to a special method consisting of a sequence of (1) Input. The algorithm receives data from a certain set.
unambiguous instructions useable by a computer for solution of a problem. (2) Output. For each set of input data, the algorithm gives the solution to the
• Informal definition of an algorithm in a computer: problem.
(3) Precision. Each instruction is clear and unambiguous.
(4) Finiteness. If we trace out the instructions of an algorithm, then for all cases,
the algorithm terminates after a finite (possibly very large) number of steps.
(5) Uniqueness. The intermediate results of each step of the algorithm are uniquely
determined and depend only on the input and the result of the previous steps.
Input Algorithm Output (6) Generality. The algorithm could be applied to solve any problem with a given
form

• Example: The problem of finding the largest integer among a number of positive
integers
• Input: the array of n positive integers a1, a2, …, an
• Output: the largest
• Example: Input 12 8 13 9 11  Output: 12 NGUYỄN KHÁNH PHƯƠNG
• Question: Design the algorithm to solve this problem SOICT– HUST

Comparing Algorithms How to Calculate Running time

• Given 2 or more algorithms to solve the same problem, how do we select the • Most algorithms transform input objects into output objects
best one?
• Some criteria for selecting an algorithm: sorting
5 3 1 2 1 2 3 5
1) Is it easy to implement, understand, modify? algorithm
input object output object
2) How long does it take to run it to completion? TIME
3) How much of computer memory does it use? SPACE • The running time of an algorithm typically grows with the input size
In this lecture we are interested in the second and third criteria: • Idea: analyze running time as a function of input size
• Even on inputs of the same size, running time can be very different
• Time complexity: The amount of time that an algorithm needs to run to
completion • Example: In order to find the first prime number in an array: the
algorithm scans the array from left to right
• Space complexity: The amount of memory an algorithm needs to run
• Array 1: 3 9 8 12 15 20 (algorithm stops when considering the first element)
We will occasionally look at space complexity, but we are mostly interested in • Array 2: 9 8 3 12 15 20 (algorithm stops when considering the 3rd element)
time complexity in this course. Thus in this course the better algorithm is the
one which runs faster (has smaller time complexity) • Array 3: 9 8 12 15 20 3 (algorithm stops when considering the last element)
 Idea: analyze running time in the
 best case

 worst case
NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST  average case SOICT– HUST

7
Kind of analyses Experimental Evaluation of Running Time
Best-case: best case • Write a program implementing the algorithm
average case
• T(n) = minimum time of algorithm on any input of size n. worst case • Run the program with inputs of varying size and composition
120

• Cheat with a slow algorithm that works fast on some input. 100
• Use a method like clock( ) to get an accurate measure of the actual running time
Average-case:

Running Time
80

• T(n) = expected time of algorithm over all inputs of size n. 60

• Need assumption of statistical distribution of inputs 40

20
• Very useful but often difficult to determine 0
• Plot the results
1000 2000 3000 4000
Worst-case: Input Size
9000

8000
• T(n) = maximum time of algorithm on any input of size n.
7000
• Easier to analyze 6000

Time (ms)
5000

4000
To evaluate the running time: 2 ways:
3000
• Experimental evaluation of running time 2000

• Theoretical analysis of running time 1000

0
0 50 100
NGUYỄN KHÁNH PHƯƠNG Input Size
SOICT– HUST

Limitations of Experiments when evaluating the running time of an algorithm Theoretical Analysis of Running Time
• Experimental evaluation of running time is very useful but • Uses a pseudo-code description of the algorithm instead of an
– It is necessary to implement the algorithm, which may be difficult implementation
• Characterizes running time as a function of the input size, n
– Results may not be indicative of the running time on other inputs
not included in the experiment • Takes into account all possible inputs
• Allows us to evaluate the speed of an algorithm independent of the
– In order to compare two algorithms, the same hardware and hardware/software environment (Changing the hardware/software
software environments must be used environment affects the running time by a constant factor, but does not
 We need: Theoretical Analysis of Running Time alter the growth rate of the running time)

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

8
Contents 1.3. Asymptotic notation
1.1. Introductory Example Q, W, O, o, w
» What these symbols do are:
1.2. Algorithm and Complexity • give us a notation for talking about how fast a function goes to infinity,
which is just what we want to know when we study the running times of
1.3. Asymptotic notation algorithms.
1.4. Running time calculation • defined for functions over the natural numbers
• used to compare the order of growth of 2 functions
Example: f(n) = Q (n2): Describes how f(n) grows in comparison to n2.
» Instead of working out a complicated formula for the exact running time, we
can just say that the running time is for example Q(n2) [read as theta of n2]:
that is, the running time is proportional to n2 plus lower order terms. For
most purposes, that’s just what we want to know.

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

𝚯 - Theta notation f(n) = Q(g(n)) c1, c2 , n0 >0 : n  n0 , c1g(n)  f(n)  c2g(n)

• For a given function g(n), we denote by 𝚯(g(n)) the set of functions Example 1: Show that 10n2 - 3n = 𝚯(n2)
 f (n) : there exist positive constants c1, c2 , and n0 s.t.
Q( g (n))    • With which values of the constants n0, c1, c2 then the inequality in the
 0  c1g ( n)  f (n)  c2 g ( n) for all n  n0 
Intuitively: Set of all functions that have the same rate of growth as g(n). definition of the theta notation is correct:
• A function f(n) belongs to the set 𝚯(g(n)) if there exist positive constants c1
and c2 such that it can be “sand- wiched” between c1g(n) and c2g(n) for 𝑐 𝑛 ≤ 𝑓(𝑛) = 10𝑛 − 3𝑛 ≤ 𝑐 𝑛 ∀n ≥ n0
sufficienly large n
• Suggestion: Make c1 a little smaller than the leading (the highest)
• 𝑓 𝑛 = 𝚯(g(n)) means that there exists some constant c1 and c2 s.t.
coefficient, and c2 a little bigger.
c1g(n) ≤ f(n) ≤ c2g(n) for large enough n.
• When we say that one function is theta of  Select: c1 = 1, c2 = 11, n0 = 1 then we have
another, we mean that neither function goes
n2 ≤ 10n2 – 3n ≤ 11n2, with n ≥ 1.
to infinity faster than the other.
 ∀n ≥ 1: 10n2 - 3n = 𝚯(n2)
• Note: For polynomial functions: To compare the growth rate, it is necessary
to look at the term with the highest coefficient

9
 f(n) = Q(g(n)) c1, c2 , n0 >0 : n  n0 , c1g(n)  f(n)  c2g(n) f(n) = Q(g(n)) c1, c2 , n0 >0 : n  n0 , c1g(n)  f(n)  c2g(n)

Example 2: Show that 𝑓 𝑛 = 𝑛 − 3𝑛 = 𝚯(𝑛 ) Example 3: Show that f(n) = 23n3 – 10 n2 log2n + 7n + 6 = 𝚯(𝑛 )
We must find n0, c1 and c2 such that
We must find n0, c1 and c2 such that
𝑐 𝑛 ≤ 𝑓(𝑛) = 𝑛 − 3𝑛 ≤ 𝑐 𝑛 ∀n ≥ n0 𝑐 n3 ≤ 𝑓(𝑛) = 23n3 – 10 n2 log2n + 7n + 6 ≤ 𝑐 n3n  n0

O - big Oh notation Graphic Illustration

For a given function g(n), we denote by O(g(n)) the set of functions
O(g(n)) = {f(n) :  positive constants c and n0, such that
 f ( n) : there exist positive constants c and n0 s.t.
O( g ( n))    n  n0, we have 0  f(n)  cg(n) }
 0  f ( n)  cg ( n) for all n  n0 
Intuitively: Set of all functions whose rate of growth is the same as or lower • f(n) = 2n+6 f(n) = 2n + 6
than that of g(n). c g(n)=4n
• Conf. def:
• We say: g(n) is asymptotic upper bound of the function f(n), to within a – Need to find a function g(n)
constant factor, and write f(n) = O(g(n)). and constants c and n0 such
• f(n) =O(g(n)) means that there exists some constant c such that f(n) is as f(n) < cg(n) when n>n0
always ≤ cg(n) for large enough n.  g(n) = n, c = 4 and n0=3
• O(g(n)) is the set of functions that go to  f(n) is O(n)
infinity no faster than g(n).
The order of f(n) is n

g(n)=n

10
Big-Oh Examples Note
• The values of positive constants n0 and c are not unique when proof the
O(g(n)) = {f(n) :  positive constants c and n0, such that
n  n0, we have 0  f(n)  cg(n) } asymptotic formulas

• Example: show that 100n + 5 = O(n2)

• Example 1: Show that 2n + 10 = O(n)
 f(n) = 2n+10, g(n) = n – 100n + 5 ≤ 100n + n = 101n ≤ 101n2 ∀n≥5
 Need constants c and n0 such that 2n + 10  cn for n  n0
n0 = 5 and c = 101 are constants need to determine
 (c  2) n  10
 n  10/(c  2) – 100n + 5 ≤ 100n + 5n = 105n ≤ 105n2 ∀ n ≥ 1
 Pick c = 3 and n0 = 10 n0 = 1 and c = 105 are also constants need to determine
• Example 2: Show that 7n-2 is O(n)
 f(n) = 7n-2, g(n) = n • Only need to find some positive constants c and n0 satisfying the equality
 Need constants c and n0 such that 7n - 2  cn for n  n0 in the definition of asymptotic notation
 (7  c ) n ≤ 2
 n ≤ 2/(7  c) NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST
 Pick c = 7 and n0 = 1

Big-Oh Examples Big-Oh Examples

O(g(n)) = {f(n) :  positive constants c and n0, such that O(g(n)) = {f(n) :  positive constants c and n0, such that
n  n0, we have 0  f(n)  cg(n) } n  n0, we have 0  f(n)  cg(n) }
1,000,000
• Example 3: Show that 3n3 + 20n2 + 5 is O(n3) n^2
• Example 5: the function n2 is 100n
Need constants c and n0 such that 3n3 + 20n2 + 5  cn3 for n  n0 not O(n)
100,000
10n
…… – n2  cn 10,000 n

• Example 4: Show that 3 log n + 5 is O(log n) – nc

1,000
Need constants c and n0 such that 3 log n + 5  clog n for n  n0 – The above inequality cannot
…… be satisfied since c must be a 100
constant
10

1
1 10 100 1,000
NGUYỄN KHÁNH PHƯƠNG n
SOICT– HUST

11
Big-Oh and Growth Rate Inappropriate Expressions
• The big-Oh notation gives an upper bound on the growth rate of a
function
• The statement “f(n) is O(g(n))” means that the growth rate of f(n) is

 O(g(n))
no more than the growth rate of g(n)
• We can use the big-Oh notation to rank functions according to their
growth rate
f (n)X
f(n) is O(g(n)) g(n) is O(f(n))

f (n)X
 O(g(n))
g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Big-Oh Examples Useful Big-Oh Rules

• If f(n) is a polynomial of degree d: f (n   a0  a1 n  a2 n  ...  ad n
2 d
• 50n3 + 20n + 4 is O(n3)
then f(n) is O(nd), i.e.,
– Would be correct to say is O(n3+n)
1. Drop lower-order terms
• Not useful, as n3 exceeds by far n, for large values 2. Drop constant factors
– Would be correct to say is O(n5) Example: 3n3 + 20n2 + 5 is O(n3)
• OK, but g(n) should be as close as possible to f(n)
• 3log(n) + log (log (n)) = O( ? ) • If f(n) = O(nk) then f(n) = O(np) with ∀ p > k
Example: 2n2 = O(n2) then 2n2 = O(n3)
When evaluate asymptotic f(n) = O(g(n)), we want to find function g(n) with a slower
growth rate as possible

•Simple Rule: Drop lower order

• Use the smallest possible class of functions
terms and constant factors Example: Say “2n is O(n)” instead of “2n is O(n2)”

• Use the simplest expression of the class

NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST
Example: Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)” NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

12
O Notation Examples Properties
• All these expressions are O(n): • If f(n) is O(g(n)) then af(n) is O(g(n)) for any a
• If f(n) is O(g1(n)) and h(n) is O(g2(n)) then
– n, 3n, 61n + 5, 22n – 5, …
• f(n)+h(n) is O(g1(n)+g2(n))
• All these expressions are O(n2): • f(n)h(n) is O(g1(n) g2(n))
– n2, 9 n2, 18 n2+ 4n – 53, … • If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))
• All these expressions are O(n log n): • If p(n) is a polynomial in n then log p(n) is O(log(n))
• If p(n) is a polynomial of degree d, then p(n) is O(nd)
– n(log n), 5n(log 99n), 18 + (4n – 2)(log (5n + 3)), …
• nx = O(an), for any fixed x > 0 and a > 1
– An algorithm of order n to a certain power is better than an algorithm of order a
( > 1) to the power of n
• log nx is O(log n), for x > 0 – how?
• logxn is O(ny) for x > 0 and y > 0
– An algorithm of order logn (to a certain power) is better than an algorithm of n
raised to a power y.

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Ω-Omega notation Omega Examples

• For a given function g(n), we denote by Ω(g(n)) the set of functions W(g(n)) = {f(n) :  positive constants c and n0, such that
 f (n) : there exist positive constants c and n0 s.t.
W( g (n))    n  n0, we have 0  cg(n)  f(n)}
 0  cg (n)  f ( n) for all n  n0 
Intuitively: Set of all functions whose rate of growth is the same as or higher • Example 1: Show that 5n2 is W(n)
than that of g(n). Need constants c and n0 such that cn  5n2 for n  n0
• We say: g(n) is asymptotic lower bound of the function f(n), to within a this is true for c = 1 and n0 = 1
constant factor, and write f(n) = Ω(g(n)).
Comment:
• f(n) = Ω(g(n)) means that there exists some constant c such that f(n) is
always ≥ cg(n) for large enough n. • If f(n) = W(nk) then f(n) = W(np) with ∀ p < k.
• Ω(g(n)) is the set of functions that go to – Example: If f(n) = W(n5) then f(n) = W(n)
infinity no slower than g(n)
• When evaluate asymptotic f(n) = W(g(n)), we want to find function
g(n) with a faster growth rate as possible

13
Asymptotic notation in equations Asymptotic notation
Another way we use asymptotic notation is to simplify calculations:
• Use asymptotic notation in equations to replace expressions
containing lower-order terms
Example:
4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + Q(n)
= 4n3 + Q(n2) = Q(n3)
Graphic examples of 𝚯, O, and Ω
How to interpret?
In equations, Q(f(n)) always stands for an anonymous function g(n)  Q(f(n))
Theorem: For any two functions f(n) and g(n), we have 𝑓 𝑛 = 𝚯(g(n))
– In this example, we use Q(n2) stands for 3n2 + 2n + 1
if and only if
𝑓 𝑛 = 𝑂(g(n)) and 𝑓 𝑛 = Ω(g(n))

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Theorem: For any two functions f(n) and g(n), we have 𝑓 𝑛 = 𝚯(g(n)) Theorem: For any two functions f(n) and g(n), we have 𝑓 𝑛 = 𝚯(g(n))
if and only if 𝑓 𝑛 = 𝑂(g(n)) and 𝑓 𝑛 = Ω(g(n)) if and only if 𝑓 𝑛 = 𝑂(g(n)) and 𝑓 𝑛 = Ω(g(n))

Example 1: Show that f(n) = 5n2 = 𝚯(𝑛 ) Example 2: Show that f(n) = 3n2 – 2n + 5 = 𝚯(𝑛 )
Because: Because:
• 5n2 = O 𝑛2 3n2 – 2n + 5 = O 𝑛2
f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n0  1 such that f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n0  1 such that
f(n) ≤ cg(n) for n  n0 f(n) ≤ cg(n) for n  n0
let c = 5 and n0 = 1  pick c = ? and n0 = ?
• 5n2 = Ω(n2) 3n2 – 2n + 5 = Ω(n2)
f(n) is W(g(n)) if there is a constant c > 0 and an integer constant n0  1 such that f(n) is W(g(n)) if there is a constant c > 0 and an integer constant n0  1 such that
f(n)  cg(n) for n  n0 f(n)  cg(n) for n  n0
let c = 5 and n0 = 1  pick c = ? and n0 = ?
Therefore: f(n) = Q(n2) Therefore: f(n) = Q(n2)

𝑓 𝑛 = 𝚯(g(n)) f(n) = O(g(n)) f(n) = Ω(g(n)) 𝑓 𝑛 = 𝚯(g(n)) f(n) = O(g(n)) f(n) = Ω(g(n))

14
Exercise 1 Exercise 2
Show that: 100n + 5 ≠ W(n2) Show that: n ≠ Q(n2)

Ans: Contradiction Ans: Contradiction

– Assume: 100n + 5 = W(n2) – Assume: n = Q(n2)

  c, n0 such that: 0  cn2  100n + 5

– We have: 100n + 5  100n + 5n = 105n  n  1

– Therefore: cn2  105n  n(cn – 105)  0

– As n > 0  cn – 105  0  n  105/c

The above inequality cannot be satisfied since c must be a constant

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

Exercise 3:Show that The way to talk about the running time
a) 6n3 ≠ Q(n2) • When people say “The running time for this algorithm is O(f(n))”, it means
Ans: Contradiction that the worst case running time is O(f(n)) (that is, no worse than c*f(n)
for large n, since big Oh notation gives an upper bound).
– Assume: 6n3 = Q(n2)
• It means the worst case running time could be determined by some
function g(n)  O(f(n))

b) n ≠ Q(log2n) • When people say “The running time for this algorithm is W (f(n))”, it means
that the best case running time is W(f(n)) (that is, no better than c*f(n) for
Ans: Contradiction
large n, since big Omega notation gives a lower bound).
– Assume: n = Q(log2n)
• It means the best case running time could be determined by some
function g(n)  W (f(n))
 g ( n) :there exist positive constants c and n0 s.t.
NGUYỄN KHÁNH PHƯƠNG W( f (n))   
SOICT– HUST  0  cf ( n)  g ( n) for all n  n0 

15
 o- Little oh notation w - Little omega notation
• For a given function g(n), we denote by o(g(n)) the set of functions • For a given function g(n), we denote by o(g(n)) the set of functions
 f (n) :there exist positive constants c and n0 s.t.  f ( n) :there exist positive constants c and n0 s.t.
o( g ( n))    w ( g (n))   
 0  f (n)  cg (n) for all n  n0   0  cg (n)  f (n) for all n  n0 
f(n) becomes insignificant relative to g(n) as n approaches infinity: f(n) becomes arbitrarily large relative to g(n) as n approaches infinity:
lim [f(n) / g(n)] = 0 lim [f(n) / g(n)] = 
n n

g(n) is an upper bound for f(n) that is not asymptotically tight. g(n) is a lower bound for f(n) that is not asymptotically tight.

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Basic functions Basic Functions

• Often appear in algorithm analysis:
– Constant  1
– Logarithmic  log2 n Which are more alike ?
– Linear  n
– N-Log-N  n log2 n
– Quadratic  n2
– Cubic  n3
– Exponential  2n
n1000 n2 2n
Let’s practice classifying functions polynomial

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

16
Basic Functions Basic functions growth rates

n logn n nlogn n2 n3 2n
Which are more alike ? 4 2 4 8 16 64 16

8 3 8 24 64 512 256

16 4 16 64 256 4,096 65,536

1000n2 3n2 2n3

32 5 32 160 1,024 32,768 4,294,967,296

64 6 64 384 4,094 262,144 1.84 * 1019

128 7 128 896 16,384 2,097,152 3.40 * 1038

quadratic 256 8 256 2,048 65,536 16,777,216 1.15 * 1077

512 9 512 4,608 262,144 134,217,728 1.34 * 10154

1024 10 1,024 10,240 1,048,576 1,073,741,824 1.79 * 10308

Algorithm Types The way to remember these notations

• Time takes to solve an instance of a
– Linear Algorithm is
• Never greater than c*n Theta f(n) = Q(g(n)) f(n) ≈ c g(n)
– Quadratic Algorithm is
• Never greater than c*n2 Big Oh f(n) = O(g(n)) f(n) ≤ c g(n)
– Cubic Algorithm is
• Never greater than c*n3 Big Omega f(n) = Ω(g(n)) f(n) ≥ c g(n)
– Polynomial Algorithm is
• Never greater than nk Little Oh f(n) = o(g(n)) f(n) ≪ c g(n)
– Exponential Algorithm is
• Never greater than cn Little Omega f(n) = ω(g(n)) f(n) ≫ c g(n)
where c & k are appropriate constants

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

17
The analogy between comparing functions and comparing numbers “Relatives” of notations
One thing you may have noticed by now is that these relations are kind • “Relatives” of the Big-Oh
of like the “<, >” relations for the numbers – W (g(n)): Big Omega – asymptotic lower bound
– Q (g(n)): Big Theta – asymptotic tight bound
fg  ab
• Big-Omega – think of it as the inverse of O(n)
– f(n) is W (g(n)) if g(n) is O(f(n))
f (n) = Q(g(n))  a = b • Big-Theta – combine both Big-Oh and Big-Omega
f (n) = O(g(n))  a  b – f(n) is Q (g(n)) if f(n) is O(g(n)) and g(n) is W (f(n))
f (n) = W(g(n))  a  b • Make the difference:
f (n) = o(g(n))  a < b – 3n+3 is O(n) and is Q (n)
– 3n+3 is O(n2) but is not Q (n2)
f (n) = w (g(n))  a > b
• Little-oh – f(n) is o(g(n)) if f(n) is O(g(n)) and f(n) is not Q (g(n))
– 2n+3 is o(n2)
– 2n + 3 is o(n) ?

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Math you need to Review Logarithms and exponentials – Bases

 Exponents: • If the base of a logarithm is changed from one constant to
a(b+c) = abac
abc = (ab)c another, the value is altered by a constant factor.
ab /ac = a(b-c)
b = a logab – Ex: log10 n * log210 = log2 n.
bc = a c*logab – Base of logarithm is not an issue in asymptotic notation.
 Logarithms:
• Exponentials with different bases differ by a exponential
x = logba is the a  b log b a factor (not a constant factor).
exponent for a = bx. log c ( ab)  log c a  log c b – Ex: 2n = (2/3)n*3n.
log b a n  n log b a
Natural log: ln a = logea log c a
log b a 
Binary log: lg a = log2a log c b
log b (1 / a )   log b a
lg2a = (lg a)2 1
log b a 
lg lg a = lg (lg a) log a b
a logb c  c logb a NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

18
Exercise Properties
• Order the following functions by their asymptotic growth rates • Transitivity (truyền ứng)
1. nlog2n f(n) = Q(g(n)) & g(n) = Q(h(n))  f(n) = Q(h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
2. log2n3
f(n) = W(g(n)) & g(n) = W(h(n))  f(n) = W(h(n))
3. n2
• Reflexivity
4. n2/5 f(n) = Q(f(n)) f(n) = O(g(n)) f(n) = W(g(n))
5. 𝟐𝒍𝒐𝒈𝟐𝒏
• Symmetry (đối xứng)
6. log2(log2n)
f(n) = Q(g(n)) if and only if g(n) = Q(f(n))
7. Sqr(log2n)
• Transpose Symmetry (Đối xứng chuyển vị)

f(n) = O(g(n)) if and only if g(n) = W(f(n))

Example: A = 5n2 + 100n, B= 3n2 + 2 . Show that A  Q(B)

Ans: A  Q(n2), n2  Q(B)  A  Q(B)
NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

Limits Exercise
• lim [f(n) / g(n)] = 0  f(n)  o(g(n)) Show that
n
1) 3n2 – 100n + 6 = O(n2)
• lim [f(n) / g(n)] <   f(n)  O(g(n))
n 2) 3n2 – 100n + 6 = O(n3)
• 0 < lim [f(n) / g(n)] <   f(n)  Q(g(n)) 3) 3n2 – 100n + 6 ≠ O(n)
n

• 0 < lim [f(n) / g(n)]  f(n)  W(g(n)) 4) 3n2 – 100n + 6 = Ω(n2)

n 5) 3n2 – 100n + 6 ≠ Ω(n3)
• lim [f(n) / g(n)] =   f(n)  w(g(n)) 6) 3n2 – 100n + 6 = Ω(n)
n

• lim [f(n) / g(n)] undefined  can’t say 7) 3n2 – 100n + 6 = 𝚯(n2)

n
8) 3n2 – 100n + 6 ≠ 𝚯(n3)
Exercise: Express functions in A in asymptotic notation using functions in B. 9) 3n2 – 100n + 6 ≠ 𝚯(n)
A B
log3(n2) log2(n3) A  Q(B)
logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)  A  Q(B)
nlg4 3lg n A  w(B)
NGUYỄN KHÁNH PHƯƠNG
alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n  A  w (B) SOICT– HUST

19
Final notes Contents
Even though in this course we focus on the
•
asymptotic growth using big-Oh notation,
Running time 1.1. Introductory Example
practitioners do care about constant factors
occasionally A 1.2. Algorithm and Complexity
• Suppose we have 2 algorithms
• Algorithm A has running time 30000n 1.3. Asymptotic notation
Algorithm B has running time 3n2 B
•
•

Asymptotically, algorithm A is better than algorithm

1.4. Running time calculation
B 10000
• However, if the problem size you deal with is always
less than 10000, then the quadratic one is faster problem size

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Running time calculation Primitive Operations

• Experimental evaluation of running time: • For theoretical analysis, we will count primitive or basic operations,
– Write a program implementing the algorithm which are simple computations performed by an algorithm
– Run the program and measure the running time
– Cons of experimental evaluation:
• It is necessary to implement the algorithm, which may be difficult could be implemented within the running time that is
• Results may not be indicative of the running time on other inputs not included in bounded above by a constant independent of the input data size.
the experiment
• In order to compare two algorithms, the same hardware and software
environments must be used • Examples of primitive operations:
 We need: Theoretical Analysis of Running Time – Evaluating an expression x2+ey
• Theoretical Analysis of Running Time: – Assigning a value to a variable cnt  cnt+1
– Uses a pseudo-code description of the algorithm instead of an implementation
– Characterizes running time as a function of the input size, n
– Indexing into an array A[5]
– Takes into account all possible inputs – Calling a method mySort(A,n)
– Allows us to evaluate the speed of an algorithm independent of the – Returning from a method return(cnt)
hardware/software environment (Changing the hardware/software environment
affects the running time by a constant factor, but does not alter the growth rate of the
running time) NGUYỄN KHÁNH PHƯƠNG
SOICT– HUST

20
Running Time Calculations: General rules Some Examples
1. Consecutive Statements: The sum of running time of each segment.
• Running time of “P; Q”, where P is implemented first, then Q, is
Time(P; Q) = Time(P) + Time(Q) , Case1: for (i=0; i<n; i++)
or if using asymptotic Theta: for (j=0; j<n; j++) O(n2)
k++;
Time(P; Q) = Q(max(Time(P), Time(Q)).
Case 2: for (i=0; i<n; i++)
2. FOR loop: The number of iterations times the time of the inside statements. k++; O(n) work followed
for i =1 to m do P(i); for (i=0; i<n; i++) by O(n2) work, is
Assume running time of P(i) is t(i), then the running time of for loop is ∑ 𝑡(𝑖)
for (j=0; j<n; j++) also O (n2)
k++;
Case 3: for (int i=0; i<n-1; i++)
3. Nested loops: The product of the number of iterations times the time of the inside
for (int j=0; j<i; j++)
statements.
for i =1 to n do
int k+=1; O(n2)
for j =1 to m do P(j);
Assume the running time of P(j) is t(j), then the running time of this nested loops is:

Running Time Calculations: General rules Characteristic statement

4. If/Else • Definition. The characteristic statement is the statement
if (condition )
S1; being executed with frequency at least as well as any
else statement in the algorithm.
S2;
• If the execution time of each statement is bounded above by
The testing time plus the larger running time of the S1 and S2. a constant, then the running time of the algorithm will be
the same size as the number of times the execution of the
characteristic statement
=> To evaluate the running time, one can count the number of
times the characteristic statement being executed

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

21
Example: Calculating Fibonacci Sequences Exercise 1: Maximum Subarray Problem
function Fibrec(n) • Fibonacci Sequence: Given an array of integers A1, A2, …, AN, find the maximum value of
if n <2 then return n; – f0=0; ∑ 𝐴
else return Fibrec(n-1)+Fibrec(n-2);
– f1=1;
– fn= fn-1 + fn-2 For convenience, the maximum subsequence sum is zero if all the
function Fibiter(n) integers are negative.
i=0;
j=1;
for k=1 to n do
j=i + j; Characteristic statement
i=j – i;
• The number of times this characteristic statement being
return j;
executed is n  The running time of Fibiter is O(n)

n 10 20 30 50 100
Fibrec 8ms 1sec 2min 21days 109years

Fibiter 0.17ms 0.33ms 0.5ms 0.75ms 1.5ms NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

Algorithm 1. Brute force Algorithm 2. Brute force with better implement

int maxSum = 0;
for (int i=0; i<n; i++) {
for (int j=i; j<n; j++) {
int sum = 0;
for (int k=i; k<=j; k++)
sum += a[k]; int maxSum = a[0];
if (sum > maxSum) for (int i=0; i<n; i++) {
maxSum = sum; int sum = 0;
} for (int j=i; j<n; j++) {
}

O(n2)
sum += a[j];
if (sum > maxSum)
maxSum = sum;
Select the statement sum+=a[k] as the characteristic statement }
 Running time of the algorithm: O(n3) }

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

22
Algorithm 3. Dynamic programming Exercise 2: Selection sort
The primary steps of dynamic programming: • Sort a sequence of numbers in ascending order
1. Divide: • Algorithm:
• Define si the value of max subarray of the array a0, a1, ..., ai , i = 0, 1, ..., n-1. – Find the smallest and move it to the first place
• Clearly, sn-1 is the solution. – Find the next smallest and move it to the second place
– Find the next smallest and move it to the 3rd place
3. Construct the final solution:
– …
• s0 = a0
void selectionSort(int a[], int n){
• Assume i > 0 and we already know the value of sk with k = 0, 1, ..., i-1. Now we need to calculate the value of si int i, j, index_min;
which is the value of max subarray of the array: for (i = 0; i < n-1; i++) {
a0, a1, ..., ai-1, ai . index_min = i;
//Find the smallest element from a[i+1] till the last element
• We see that: the max subarray of this array a0, a1, ..., ai-1, ai could either include the element ai or not include for (j = i+1; j < n; j++)
the element ai  therefore, the max subarray of the array a0, a1, ..., ai-1, ai could only be one of these 2 arrays: if (a[j] < a[index_min]) index_min = j;
//move the element a[index_min] to the ith place:
– The max subarray of the array a0, a1, ..., ai-1 swap(a[i], a[index_min]);
– The max subarray of the array a0, a1, ..., ai ending at ai. }
}
 Thus, we have si = max {si-1, ei}, i = 1,2, …, n-1.
where ei is the value of the max subarray a0, a1, ..., ai ending at ai. void swap(int &a,int &b)
To calculate ei, we could use the recursive relation: {
int temp = a;
– e0 = a0; a = b;
– ei = max {ai, ei-1 + ai}, i = 1,2, ..., n-1. b = temp;
}

Exercise 3 Exercise 4
• Give asymptotic big-Oh notation for the running time T(n) • Give asymptotic big-Oh notation for the running time T(n) of the following
statement segment:
of the following statement segment:
for (int i = 1; i<=n; i++) a) int x = 0;
for (int j = 1; j<= i*i*i; j++) for (int i = 1; i <=n; i *= 2)
for (int k = 1; k<=n; k++)
x=x+1;
x = x + 1;
• Ans:
• Ans: int x = 0;
for (int i = n; i > 0; i /= 2)
x=x+1;
• Ans:

NGUYỄN KHÁNH PHƯƠNG NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST SOICT– HUST

23
Exercise 5
Give asymptotic big-Oh notation for the running time T(n) of the following
statement segment:
int n;
if (n<1000)
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "Hello\n";
else
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "world!\n";

Ans:
• T(n) is the constant when n<1000. T(n) = O(n2).

NGUYỄN KHÁNH PHƯƠNG

SOICT– HUST

24