Course WeChat Group
Algorithm Design and
Analysis
Who are we？
▪ Instructor
– Biaoshuai Tao (陶表帅)
▪
[email protected] QQ: 516481722 WeChat: abeliangroup
▪ https://jhc.sjtu.edu.cn/~bstao
▪ Teaching Assistants
Wentao He Jiarong Jin Panfeng Liu
何文韬 金佳绒 刘盼凤
Grading Policy
▪ Homework: 𝟕𝟎%
– 10 (6 writing + 4 programming) homework: 5% × 10 = 50%
– 1 midterm exam (in-class): 20%
▪ 1 final exam: 𝟑𝟎%
Homework Assignments
▪ Online Submission, LaTeX is recommended (template available on Canvas).
▪ Solution must be written in English.
▪ Collaboration is allowed, but…
– Solutions must be written individually.
– You must write down your collaborators’ names.
▪ If you have referred to external reference, you must cite it.
▪ Use of Large Language Models (LLMs) is strictly prohibited.
▪ Start it early! Do not wait until the last minute!
▪ Late submission policy:
– Less than 3 days: 50% discount
– More than 3 days: 0 credit
Coding Assignments
▪ You should submit your own code to solve it.
– All submissions in your account counts. If you help others
debugging, do not submit using your own account.
– Protecting your solution from copying is your
responsibility. Please, take reasonable steps to protect your
work. We strongly suggest you not to share code with others before
the end of the term.
– If your behavior is not distinguishable from plagiarism, and you can
not prove that you solve the task on your own, it will be considered
plagiarism.
▪ Use of Large Language Models (LLMs) is strictly prohibited.
▪ We will review code with you before making decision. If
you code it yourself, don't worry.
关于补考、缓考
▪ 原则上不安排
▪ 实在有需求之同学需征得我之同意
▪ 强烈不推荐该选项
Textbook (optional)
Algorithms
▪ by Dasgupta, Papadimitriou, Vazirani
Algorithms
▪ An algorithm is a finite sequence of well-defined
instructions for solving a problem.
Algorithms – Grade School + − × ÷
Algorithms – Find the shortest path
Algorithms – Matching
Algorithms are ubiquitous!
How to choose
data in the
cache?
Operating Systems
How to design
an encoding
algorithm?
Cryptography
Algorithm Design and Analysis
Design
▪ Intuitions
▪ Experiences
Analysis
▪ Does the algorithm work?
– Return the optimal/correct answer
– Return the solution that satisfies the given requirement
▪ Is it fast?
– Time complexity
– Worst case
▪ Can I do better?
– More efficient
– Better time complexity
How to think?
Listen to my idea, it is quite
• What is work? intuitive! It should work if
• What is better? everything goes well, trust me!
• Do we need to consider worst
case?
• Is there any corner case?
▪ Detail-oriented ▪ Big-picture
▪ Precise ▪ Intuitive
▪ Rigorous ▪ Hand-waving
Course Objectives
AI2615 – Design and Analysis of Algorithms
▪ Learn how to design algorithms
– Equip with an algorithmic toolkit and use it correctly
– Clearly communicate your algorithmic ideas
▪ Learn how to analyze your algorithms
– Correctness proof (Math Maturity)
– Time complexity
▪ Improve English skill
Roadmap
Divide and Conquer
Graph Algorithms
Greedy
Basic Complexity Dynamic
NP-hardness Max Flow
Programming
Linear Programming
Advanced Topics
Fibonacci Numbers
Computing Fibonacci Numbers
Asymptotic Notations 𝑂, Ω, Θ, 𝑜, 𝜔
Fibonacci Numbers
▪ 𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2
▪ 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, ….
Question
▪ Design an algorithm for the problem below:
▪ Input: 𝑛
▪ Output: 𝐹𝑛
A Recursive Algorithm based on Definition
▪ Pseudocode: clearly communicate algorithmic ideas
function fib1(n)
if 𝑛 = 0: return 0
if 𝑛 = 1: return 1
return fib1(𝑛 − 1) + fib1(𝑛 − 2)
Analysis
1. Is the algorithm correct?
2. How much time does it take, as a function of 𝑛?
3. Can we do better?
Analysis
1. Is the algorithm correct?
⚫ Yes, by definition!
2. How much time does it take, as a function of 𝑛?
3. Can we do better?
How much time does it take?
▪ 𝑇(𝑛): number of steps needed to compute 𝐹𝑛 .
▪ 𝑇 0 =1
▪ 𝑇 1 =2
▪ 𝑇 𝑛 =𝑇 𝑛−1 +𝑇 𝑛−2 +3
▪ What can we say about 𝑇 𝑛 ?
Prove 𝑇 𝑛 > 𝐹𝑛 by Induction
▪ Base Step: 𝑇 0 = 1 > 0 = 𝐹0 ; 𝑇 1 = 2 > 1 = 𝐹1
▪ Inductive Step:
▪ Induction Hypothesis: Suppose 𝑇 𝑖 > 𝐹𝑖 for 𝑖 = 1, … , 𝑛.
▪ We need to show 𝑇 𝑛 + 1 > 𝐹𝑛+1 .
▪ 𝑇 𝑛+1 =𝑇 𝑛 +𝑇 𝑛−1 +3
▪ > 𝐹𝑛 + 𝐹𝑛−1 + 3 (induction hypothesis)
▪ = 𝐹𝑛+1 + 3 > 𝐹𝑛+1
𝑇 𝑛 > 𝐹𝑛
▪ 𝐹𝑛 ≈ 20.694𝑛 ≈ 1.6𝑛 grows exponentially!
▪ 𝑇 𝑛 > 1.6𝑛 grows exponentially.
▪ Computing 𝐹200 takes more than 296 seconds (which is
more than 157,019,284,536,451,074,949 years) by one of
the fastest computer in the world.
▪ Unacceptably inefficient…
Analysis
1. Is the algorithm correct?
⚫ Yes, by definition!
2. How much time does it take, as a function of 𝑛?
⚫ Exponentially slow!
3. Can we do better?
Proliferation of Recursive Calls
𝐹𝑛
𝐹𝑛−1 𝐹𝑛−2
𝐹𝑛−2 𝐹𝑛−3 𝐹𝑛−3 𝐹𝑛−4
𝐹𝑛−3 𝐹𝑛−4 𝐹𝑛−4 𝐹𝑛−5 𝐹𝑛−4 𝐹𝑛−5 𝐹𝑛−5 𝐹𝑛−6
Many Repeated Computations…
𝐹𝑛
𝐹𝑛−1 𝐹𝑛−2
𝐹𝑛−2 𝐹𝑛−3 𝐹𝑛−3 𝐹𝑛−4
𝐹𝑛−3 𝐹𝑛−4 𝐹𝑛−4 𝐹𝑛−5 𝐹𝑛−4 𝐹𝑛−5 𝐹𝑛−5 𝐹𝑛−6
We should store those intermediate values once computed to avoid repeated
computations!
The second algorithm
function fib2(n)
if 𝑛 = 0: return 0
create an array 𝑓[0 … 𝑛]
𝑓 0 = 0, 𝑓 1 = 1
for 𝑖 = 2, … , 𝑛:
𝑓 𝑖 = 𝑓 𝑖 − 1 + 𝑓[𝑖 − 2]
return 𝑓[𝑛]
How much time does it take?
▪ It takes about 𝑛 “steps”.
▪ From exponential time to polynomial time
▪ Now we can easily compute 𝐹200 , or even 𝐹2000000 .
A Caveat
▪ We have assume 𝑓 𝑖 = 𝑓 𝑖 − 1 + 𝑓[𝑖 − 2] takes one step.
▪ Can we assume addition takes constant amount of time?
▪ This is reasonable for not-very-large numbers (e.g., on a 64-bit
machine).
▪ However, 𝐹𝑛 ≈ 20.694𝑛 grows exponentially fast.
▪ 𝐹100 already needs more than 69 bits to encode.
▪ We are indeed adding two large numbers in each iteration!
Computational Models
𝑞start
0 → 1, 𝑅 1 → 1, 𝑅
1 → 1, 𝑅
𝑞1 𝑞2
1 → 0, 𝑅
0 → 1, 𝐿 0 → 0, 𝑅
𝑞acc 𝑞rej
0 1 1 0 0 0 1 0 1 …
Turing Machine Word RAM
Real Running Time for fib2(n)
▪ Adding two 𝑛-bit numbers takes about 𝑛 units of time.
▪ What is the time complexity for fib2(n)?
▪ A. of order 𝑛 B. of order 𝑛1.5 C. of order 𝑛2
Asymptotic Notations
Read Notation
Big-O 𝑂(⋅)
Big-Omega Ω(⋅)
Big-Theta Θ(⋅)
Little-o 𝑜(⋅)
Little-omega 𝜔(⋅)
Big-O (asymptotic ≤)
▪ 𝑓 𝑛 = 𝑂 𝑔 𝑛 : there exist 𝑛0 , 𝐶 ∈ ℤ+ such that 𝑓 𝑛 ≤ 𝐶𝑔(𝑛)
for all 𝑛 ≥ 𝑛0 .
▪ A sufficient (but not necessary) condition:
𝑓 𝑛
lim <∞
𝑛→∞ 𝑔 𝑛
▪ 𝑓(𝑛) is asymptotically no greater than 𝑔(𝑛).
Big-Omega (asymptotic ≥)
▪ 𝑓 𝑛 = Ω 𝑔 𝑛 : there exist 𝑛0 , 𝐶 ∈ ℤ+ such that 𝑓 𝑛 ≥ 𝐶𝑔(𝑛)
for all 𝑛 ≥ 𝑛0 .
▪ A sufficient (but not necessary) condition:
𝑓 𝑛
lim >0
𝑛→∞ 𝑔 𝑛
▪ 𝑓(𝑛) is asymptotically no smaller than 𝑔(𝑛).
Big-Theta (asymptotic =)
▪ 𝑓 𝑛 = Θ 𝑔 𝑛 : we have both 𝑓 𝑛 = 𝑂 𝑔 𝑛 and 𝑓 𝑛 =
Ω 𝑔 𝑛 .
▪ A sufficient (but not necessary) condition:
𝑓 𝑛
∞ > lim >0
𝑛→∞ 𝑔 𝑛
▪ 𝑓(𝑛) is asymptotically equal to 𝑔(𝑛).
Little-o (asymptotic <)
▪ 𝑓 𝑛 = 𝑜 𝑔 𝑛 : for any 𝜖 > 0, there exist 𝑛0 ∈ ℤ+ such that
𝑓 𝑛 ≤ 𝜀 ⋅ 𝑔(𝑛) for all 𝑛 ≥ 𝑛0 .
▪ A necessary and sufficient condition:
𝑓 𝑛
lim =0
𝑛→∞ 𝑔 𝑛
▪ 𝑓(𝑛) is asymptotically smaller than 𝑔(𝑛).
▪ 𝑓 𝑛 =𝑜 𝑔 𝑛 ⟹𝑓 𝑛 =𝑂 𝑔 𝑛
Little-omega (asymptotic >)
▪ 𝑓 𝑛 = 𝜔 𝑔 𝑛 : for any 𝜖 > 0, there exist 𝑛0 ∈ ℤ+ such that
𝑓 𝑛 ≥ 𝜀 ⋅ 𝑔(𝑛) for all 𝑛 ≥ 𝑛0 .
▪ A necessary and sufficient condition:
𝑓 𝑛
lim =∞
𝑛→∞ 𝑔 𝑛
▪ 𝑓(𝑛) is asymptotically greater than 𝑔(𝑛).
▪ 𝑓 𝑛 =𝜔 𝑔 𝑛 ⟹𝑓 𝑛 =Ω 𝑔 𝑛
Tips
▪ Multiplicative constants can be omitted:
– 14𝑛2 = Θ 𝑛2
▪ 𝑛𝑎 dominates 𝑛𝑏 for 𝑎 > 𝑏:
1 3
– 𝑛 = 𝜔 𝑛2
12
▪ Exponentials dominate polynomials:
– 1.3𝑛 = 𝜔 𝑛12
– 0.5𝑛 = 𝑜 𝑛−2
▪ Polynomials dominates logarithms:
– 𝑛0.1 = 𝜔 log 𝑛 10
Sequences
▪ Below, 𝑎 > 0, 𝑏 ≥ 0, 𝑝 > 0 are constants (i.e., do not depend on 𝑛)
▪ Arithmetic sequence:
𝑛
(𝑛 + 1)(𝑎𝑛 + 2𝑏)
𝑎𝑖 + 𝑏 = = Θ 𝑛2
2
𝑖=0
▪ Geometric sequence:
𝑛 Θ 1 if 𝑝 ∈ (0,1)
𝑎 ⋅ 𝑝𝑖 = ቐ Θ 𝑛 if 𝑝 = 1
𝑖=0 Θ 𝑝𝑛 if 𝑝 > 1
▪ Harmonic sequence:
𝑛
1
= Θ log 𝑛
𝑖
𝑖=1
Exercise
Decide A) 𝑓 𝑛 = 𝜔 𝑔 𝑛 B) 𝑓 𝑛 = Θ 𝑔 𝑛 C) 𝑓 𝑛 = 𝑜 𝑔 𝑛
for each of the followings.
𝑛2 2
1. 𝑓 𝑛 = log 𝑛
𝑔 𝑛 = 𝑛 log 𝑛
2. 𝑓 𝑛 = log 2𝑛 𝑔 𝑛 = log 3𝑛
3. 𝑓 𝑛 = 𝑛 ⋅ 2𝑛 𝑔 𝑛 = 3𝑛
4. 𝑓 𝑛 = log 𝑛 log 𝑛 𝑔 𝑛 =2 log 𝑛 2
5. 𝑓 𝑛 = 𝑛! 𝑔 𝑛 = 2𝑛
Time Complexity for fib1 and fib2
▪ fib1(n): Θ 𝑐 𝑛 for some 𝑐 ≈ 1.6
▪ fib2(n): Θ 𝑛2
▪ Can we do better?
▪ Yes!
General Formula
𝑛 𝑛
1 1+ 5 1− 5
𝐹𝑛 = −
5 2 2
▪ Can we directly compute 𝐹𝑛 using this?
▪ No, computer cannot handle irrational numbers.
Bypass the irrational numbers barrier…
𝐹𝑛 = 𝐹𝑛
▪ Write ቊ in terms of matrix operation below:
𝐹𝑛+1 = 𝐹𝑛 + 𝐹𝑛−1
𝐹𝑛 0 1 𝐹𝑛−1
=
𝐹𝑛+1 1 1 𝐹𝑛
▪ Then we have
𝑛 𝑛
𝐹𝑛 0 1 𝐹0 0 1 0
= =
𝐹𝑛+1 1 1 𝐹1 1 1 1
▪ It only remains to compute
𝑛
0 1 0
1 1 1
𝑛
𝐹𝑛 0 1 0
fib3(n): compute =
𝐹𝑛+1 1 1 1
▪ We only need 𝑂 log 𝑛 matrix multiplications. Why?
▪ Each matrix multiplications requires 4 additions and 8
multiplications of integers. Why?
▪ Let 𝑀(𝑛) be the time complexity for multiplying two 𝑛-bit
integers.
▪ Can you show that fib3(n) has time complexity 𝑂 𝑀 𝑛 log 𝑛 ?
▪ By a more careful analysis, fib3(n) has time complexity
𝑂 𝑀 𝑛 . Do you see why?
Time Complexity for Integer Multiplications
▪ fib3(n) has time complexity 𝑂 𝑀 𝑛 .
▪ Naïve grade school multiplication yields 𝑀 𝑛 = 𝑂 𝑛2 .
▪ Future lecture: 𝑀 𝑛 = 𝑂 𝑛 log 𝑛
𝑛 𝑛 𝑛
𝐹𝑛 0 1 0 1 1+ 5 1− 5
= is just another way for computing −
𝐹𝑛+1 1 1 1 5 2 2
0 1 1+ 5 1− 5
▪ Eigenvalues for 𝐴 = :𝜆 = , 𝜆2 =
1 1 1 2 2
0 1 1 1
▪ Eigenvectors for 𝐴 = : 𝐯1 = , 𝐯2 =
1 1 𝜆 1 𝜆2
𝜆1 0
▪ Diagonalization: 𝐴 = 𝑃Λ𝑃−1 where 𝑃 = 𝐯1 𝐯2 , Λ =
0 𝜆2
▪ Then
𝐹𝑛 0 1
𝑛
0 0 𝜆𝑛
0 −1 0 1 𝜆1𝑛 − 𝜆𝑛2
= = 𝑃Λ𝑛 𝑃−1 =𝑃 1 𝑃 =
𝐹𝑛+1 1 1 1 1 0 𝜆𝑛2 1 𝑛+1
5 𝜆1 − 𝜆2
𝑛+1
Believe me about the last step…
Today we have seen…
▪ Course Introduction
▪ Fibonacci Numbers Computation
– Three algorithms
– Asymptotic notations
