Dept.

of ECE, TKU
Fall, 2023

Opportunities and Challenges of The AI Era

Chapter 2

Recurrence & Divide-and-Conquer

© 2023 by Jiann-Chyi Rau

Dept. of ECE, TKU
Fall, 2023

Recurrence
• A recurrence is an equation (or inequality) that
describes a function in terms of its values on
smaller inputs
• Cost model
T(n) = (1), if n c
T(n) = D(n) + aT(n/b) + C(n), otherwise
• a, b, and c are constants
• D(n) is the cost (time) for dividing the problem into a
subproblems with 1/b size of the original one
• C(n) is the cost (time) for combining the solutions to the
subproblems into the solutions to the original problem
© 2023 by Jiann-Chyi Rau Algorithms 2-2
Dept. of ECE, TKU
Fall, 2023

Divide-and-Conquer (DC)
• Divide the problem into a number of
subproblems that are smaller instances of
the same problem
• Conquer the subproblems by solving them
recursively
– If the subproblem sizes are small enough (base
cases), solve them in a straightforward manner
• Combine the solutions to the subproblems
into the solution for the original problem

© 2023 by Jiann-Chyi Rau Algorithms 2-3

Dept. of ECE, TKU
Fall, 2023

Three Problems (I)

• Merge Sort
• Maximum-subarray
• Strassen’s algorithm for matrix
multiplication

© 2023 by Jiann-Chyi Rau Algorithms 2-4

Dept. of ECE, TKU
Fall, 2023

Merge Sort (1/2)

• Input A = [5, 2, 4, 7, 1, 3, 2, 6]
– Divide [5, 2, 4, 7][1, 3, 2, 6]
– Divide [5, 2][4, 7][1, 3][2, 6]
– Divide [5][2][4][7][1][3][2][6]
– Merge, Merge, Merge

//check for the base case (a single element)

//divide
//conquer
//conquer
//combine
© 2023 by Jiann-Chyi Rau Algorithms 2-5
Dept. of ECE, TKU
Fall, 2023

Merge Sort (2/2)

© 2023 by Jiann-Chyi Rau Algorithms 2-6

Dept. of ECE, TKU
Fall, 2023

MERGE Process (1/3)

// sentinel card in L
// sentinel card in R

p q q+1 r
L R

© 2023 by Jiann-Chyi Rau Algorithms 2-7

Dept. of ECE, TKU
Fall, 2023

MERGE Process (2/3)

(a) Call Merge(A, 9, 12, 16) (b)

(c) (d)

(e) (f)
© 2023 by Jiann-Chyi Rau Algorithms 2-8
Dept. of ECE, TKU
Fall, 2023

MERGE Process (3/3)

(g) (h)

(i)

© 2023 by Jiann-Chyi Rau Algorithms 2-9

Dept. of ECE, TKU
Fall, 2023

Prove Merge Sort Correctness

• The key operation of the merge sort
algorithm is the merging of two “sorted”
sequences in the “Combine” step
– MERGE(A, p, q, r) merges the two “sorted”
subarrays A[p…q] and A[q+1…r] to form a
single “sorted” subarray that replaces the
current subarray A[p…r]

© 2023 by Jiann-Chyi Rau Algorithms 2-10

Dept. of ECE, TKU
Fall, 2023

Why Is MERGE Linear Time? (1/2)

• Think of two piles of cards
– Each pile is sorted and placed face-up with the
smallest card on top
– We will merge these into a single sorted pile

© 2023 by Jiann-Chyi Rau Algorithms 2-11

Dept. of ECE, TKU
Fall, 2023

Why Is MERGE Linear Time? (2/2)

• Perform the following basic operations
1.Choose and remove the smaller of the two top cards
2.Place the chosen card face-down onto the output pile
3.Repeat 1. and 2. until one input pile is empty, then
take the remaining input pile and place it face-down
onto output pile
Each step should take constant time for at most n
cards, therefore, this procedure takes linear time

© 2023 by Jiann-Chyi Rau Algorithms 2-12

Dept. of ECE, TKU
Fall, 2023

Time Complexity T(n) of Merge Sort

• Merge sort is a recursive function
T(n)
(1)
(1)
T(n/2)
T(n/2)
(n)

• T(n) can be calculated recursive

T(n) = (1), if n = 1
T(n) = (1) + 2T(n/2) + (n)
= 2T(n/2) + (n), if n > 1
© 2023 by Jiann-Chyi Rau Algorithms 2-13
Dept. of ECE, TKU
Fall, 2023

Solve T(n) Using A Recursion Tree

Assume n is an exact power of 2

 T(n) = cn(lgn+1), c is a constant and lg means log2
 T(n) = Θ(nlgn)
© 2023 by Jiann-Chyi Rau Algorithms 2-14
Dept. of ECE, TKU
Fall, 2023

Insertion Sort vs. Merge Sort

• Let input size n = 1,000,000
– Case 1: Insertion sort takes 2n2 instructions on
the supercomputer with 100 MIPS*
• Spend about 5.56 hours
– Case 2: Merge sort takes 50nlgn instructions on
the PC with 1 MIPS
• Spend 16.67 minutes (efficient!!)

*MIPS: Million Instructions Per Second

© 2023 by Jiann-Chyi Rau Algorithms 2-15
Dept. of ECE, TKU
Fall, 2023

Food for Thought from T(n)

• Θ(nlgn) is smaller than Θ(n2)
– Merge sort is faster than insertion sort
Q: Merge sort is weaker than insertion sort in
one thing…
Why don’t we use merge sort when we sort
card?
No free lunch (space issue…)

© 2023 by Jiann-Chyi Rau Algorithms 2-16

Dept. of ECE, TKU
Fall, 2023

Problem：逆序對數
• 給一數列a1, a2, ..., an，請設計一個O(nlgn)
演算法去求它的逆序對數，即有多少個有
序對(i, j)，使得i < j且 ai > aj。
– 提示：Merge Sort

© 2023 by Jiann-Chyi Rau Algorithms 2-17

Dept. of ECE, TKU
Fall, 2023

Three Problems (II)

• Merge Sort
• Maximum-subarray
• Strassen’s algorithm for matrix
multiplication

© 2023 by Jiann-Chyi Rau Algorithms 2-18

Dept. of ECE, TKU
Fall, 2023

Stock Profit (1/4)

• To maximize profit
– Buy at the lowest “possible” price and sell at
the height “possible” price

H impossible!!

© 2023 by Jiann-Chyi Rau Algorithms 2-19

Dept. of ECE, TKU
Fall, 2023

Stock Profit (2/4)

∆1 = 13
∆2 = 43
H

∆1 = 13 < ∆2 = 43  buy after day 7 and sell after day 11

© 2023 by Jiann-Chyi Rau Algorithms 2-20
Dept. of ECE, TKU
Fall, 2023

Stock Profit (3/4)

∆1 = 1
H
∆= 3

counterexample

© 2023 by Jiann-Chyi Rau Algorithms 2-21

Dept. of ECE, TKU
Fall, 2023

Stock Profit (4/4)

• A brute-force solution
– Try every possible pair of buy and sell dates in
which the buy date precedes the sell date
n
– A period of n days has  2  such pairs of dates,
 
the time complexity would be (n2)
• Can we do better?

© 2023 by Jiann-Chyi Rau Algorithms 2-22

Dept. of ECE, TKU
Fall, 2023

The Maximum-Subarray Problem

• Input: An array A[1…n] of numbers
• Output: Indices i and j such that
– A[i…j] has the greatest sum of contiguous
subarrays of A, along with the sum of the values in
A[i…j]
• E.g. Given A = [1, -4, 2, 3, 4, -6, -7]
i = 3 and j = 5 such that subarray [2, 3, 4] has the
greatest sum 9

© 2023 by Jiann-Chyi Rau Algorithms 2-23

Dept. of ECE, TKU
Fall, 2023

Possible Locations of Subarrays

• Let mid be the midpoint of the array A[low…high]
• Any contiguous subarray A[i…j] of A[low…high]
must lie in exactly one of the following places
– Entirely in the subarray A[low…mid]
– Entirely in the subarray A[mid+1…high]
– Cross the midpoint

© 2023 by Jiann-Chyi Rau Algorithms 2-24

Dept. of ECE, TKU
Fall, 2023

The Divide-and-Conquer Policy

• Divide
– Split a array into two subarrays of as equal size
as possible
• Conquer
– Find the maximum subarrays in three possible
locations
• Combine
– Choose the best solution among the three
locations
© 2023 by Jiann-Chyi Rau Algorithms 2-25
Dept. of ECE, TKU
Fall, 2023

FIND-MAX-CROSSING-SUBARRAY(A, low, mid, high)

• Time complexity: Θ(n)

© 2023 by Jiann-Chyi Rau Algorithms 2-26

Dept. of ECE, TKU
Fall, 2023

FIND-MAXIMUM-SUBARRY(A, low, high)

• Time complexity:
T(n) = 2T(n/2)+Θ(n)  T(n) = Θ(nlgn)

θ(1)
Divide θ(1)
T(n/2)
Conquer T(n/2)
θ(n)

Combine θ(1)

© 2023 by Jiann-Chyi Rau Algorithms 2-27

Dept. of ECE, TKU
Fall, 2023

Better Solution for The MS Problem

• Kadane's algorithm
– Incremental approach
– Let prefix-sum Sj = A1+A2+…+Aj , then

Ai+Ai+1+…+Aj = Sj – Si-1

– O(n)

© 2023 by Jiann-Chyi Rau Algorithms 2-28

Dept. of ECE, TKU
Fall, 2023

Three Problems (III)

• Merge Sort
• Maximum-subarray
• Strassen’s algorithm for matrix
multiplication

© 2023 by Jiann-Chyi Rau Algorithms 2-29

Dept. of ECE, TKU
Fall, 2023

Matrix Multiplication
• Input: Two n x n (square) matrices, A = (aij)
and B = (bij)
• Output: n x n matrix C = (cij) where C = AB

• Brute-force algorithm
– T(n) = Θ(n3)

© 2023 by Jiann-Chyi Rau Algorithms 2-30

Dept. of ECE, TKU
Fall, 2023

Simple Divide-and-Conquer (1/3)

• Partition A and B into eight n/2 x n/2 matrices
(assume that n is an exact power of 2)
 A11 A12   B11 B12 
A    B   
 A21 A22   B21 B22 
 C11 C12   A11 A12   B11 B12 
C         
 C21 C22   A21 A22   B21 B22 

© 2023 by Jiann-Chyi Rau Algorithms 2-31

Dept. of ECE, TKU
Fall, 2023

Simple Divide-and-Conquer (2/3)

• Complexity analysis
– Partition is Θ(1) (index calculation instead of
copying)
– Recursive calls are 8T(n/2)
– Four n2/4 additions, Θ(n2)
T(n) = 8T(n/2) + Θ(n2)
Totally, still Θ(n3), proofed later

© 2023 by Jiann-Chyi Rau Algorithms 2-32

Dept. of ECE, TKU
Fall, 2023

Simple Divide-and-Conquer (3/3)

© 2023 by Jiann-Chyi Rau Algorithms 2-33

Dept. of ECE, TKU
Fall, 2023

Strassen’s Algorithm (1/5)

• Benefits
– Perform 7 rather than 8 recursive
multiplications of n/2 x n/2 matrices
• Four steps
– Step 1: Partition into four n/2 x n/2 matrices
• Take Θ(1) time

© 2023 by Jiann-Chyi Rau Algorithms 2-34

Dept. of ECE, TKU
Fall, 2023

Strassen’s Algorithm (2/5)

– Step 2: Calculate 10 S matrices
• Take Θ(n2) time

© 2023 by Jiann-Chyi Rau Algorithms 2-35

Dept. of ECE, TKU
Fall, 2023

Strassen’s Algorithm (3/5)

– Step 3: Calculate 7 P matrices
• Take 7T(n/2) time

© 2023 by Jiann-Chyi Rau Algorithms 2-36

Dept. of ECE, TKU
Fall, 2023

Strassen’s Algorithm (4/5)

– Step 4: Construct the 4 n/2 x n/2 submatrices of
the product C (8 n/2 x n/2 matrix
addition/subtraction)
• Take Θ(n2) time

C11=P5+P4-P2+P6

© 2023 by Jiann-Chyi Rau Algorithms 2-37

Dept. of ECE, TKU
Fall, 2023

Strassen’s Algorithm (5/5)

C12=P1+P2

C21=P3+P4

C22=P5+P1-P3-P7

© 2023 by Jiann-Chyi Rau Algorithms 2-38

Dept. of ECE, TKU
Fall, 2023

Analysis of Strassen’s Algorithm

• Strassen’s complexity

T(n) = Θ(nlg7) = Θ(n2.808), proofed later

• Strassen’s algorithm was the first to beat n3
time, but not fastest
– A method by Coppersmith and Winograd runs
in O(n2.376) time
– In 2014, François Le Gall reduced to O(n2.373)
© 2023 by Jiann-Chyi Rau Algorithms 2-39
Dept. of ECE, TKU
Fall, 2023

Issues against Strassen’s Algorithm

• Higher constant factor than the brute-force
Θ(n3) method makes it impractical unless
n  45
• Not good for sparse matrices
– A sparse matrix is one with many zero entries
• Derived matrices Si and Pj consume space
Numerically unstable
– Larger errors can accumulate than brute-force
method

© 2023 by Jiann-Chyi Rau Algorithms 2-40

Dept. of ECE, TKU
Fall, 2023

Three Methods for Solving Recurrence

• Substitution: Guess and prove by

mathematical induction
• Recursion tree: Convert the recurrence into
a tree
• Master theorem: Prove bounds for the
general form
T(n) = aT(n/b) + f(n)

© 2023 by Jiann-Chyi Rau Algorithms 2-41

Dept. of ECE, TKU
Fall, 2023

Substitution
• Upper bound
• Lower bound
• Tight bound

© 2023 by Jiann-Chyi Rau Algorithms 2-42

Dept. of ECE, TKU
Fall, 2023

Substitution for Upper Bound (1/2)

• Consider
1 if n  1
T ( n)  
2T ( n / 2)  n if n  1

– Guess T(n) = O(nlgn) (upper bound)

– Prove that T(n) ≤ cnlgn, for some c > 0 (must
find c)

© 2023 by Jiann-Chyi Rau Algorithms 2-43

Dept. of ECE, TKU
Fall, 2023

Substitution for Upper Bound (2/2)

• Proof
1. Basis
• T(1) = 1 does not hold, try next
• T(2) = 4 ≤ c2lg2, true for c  2 (as n0 = 2)
2. Mathematical induction
• Assume T(m) ≤ cmlgm , T (n)  2(c n / 2 lg( n / 2))  n
for m < n  cn lg(n / 2)  n
• Let m  n / 2 ,  cn lg n  cn lg 2  n
T ( n / 2 )  c n / 2 lg ( n / 2)  cn lg n  cn  n
 cn lg n, true for c  1
© 2023 by Jiann-Chyi Rau Algorithms 2-44
Dept. of ECE, TKU
Fall, 2023

Substitution for Lower Bound (1/2)

• Consider
1 if n  1
T ( n)  
2T ( n / 2)  n if n  1

– Guess T(n) =  (nlgn) (lower bound)

– Prove that T(n)  dnlgn, for some d > 0 (must
find d)

© 2023 by Jiann-Chyi Rau Algorithms 2-45

Dept. of ECE, TKU
Fall, 2023

Substitution for Lower Bound (2/2)

• Proof
1. Basis
• T(1) = 1 > d·0, true for any d > 0 (as n0 = 1)
2. Mathematical induction
• Assume T(m)  dmlgm, T (n)  2(d (n / 2) lg(n / 2))  n
for m < n,
 dn lg(n / 2)  n
• Let m  n / 2
 dn lg n  dn lg 2  n
T ( n / 2 )  d n / 2 lg ( n / 2)  dn lg n  dn  n
 dn lg n, true for 0  d  1
© 2023 by Jiann-Chyi Rau Algorithms 2-46
Dept. of ECE, TKU
Fall, 2023

Substitution for Tight Bound

• Cosinder
1 if n  1
T ( n)  
2T ( n / 2)  n if n  1

• Since T(n) = O(nlgn) and T(n) =  (nlgn)

T(n) = Θ(nlgn)

© 2023 by Jiann-Chyi Rau Algorithms 2-47

Dept. of ECE, TKU
Fall, 2023

Recursion Tree
• Each node represents cost of a single
subproblem
– E.g. T(n) = T(n/3) + T(2n/3) + cn

(2/3)kn = 1
k = log3/2n

© 2023 by Jiann-Chyi Rau Algorithms 2-48

Dept. of ECE, TKU
Fall, 2023

Master Theorem

© 2023 by Jiann-Chyi Rau Algorithms 2-49

Dept. of ECE, TKU
Fall, 2023

Example (I) of Master Theorem

• Matrix multiplication: T(n) = 8T(n/2) + Θ(n2)
– a = 8, b = 2, and f(n) = Θ(n2)
– nlogb a = nlog2 8 = n3
– f(n) = O(nlog 8-), where  =1
2

– Apply case 1, T(n) = (n3)

© 2023 by Jiann-Chyi Rau Algorithms 2-50

Dept. of ECE, TKU
Fall, 2023

Example (II) of Master Theorem

• Strassen’s algorithm: T(n) = 7T(n/2) + Θ(n2)
– a = 7, b = 2, and f(n) = Θ(n2)
– nlog a = nlog 7 = n2.808
b 2

– f(n) = O(nlog 7-), where  =0.8

2

– Apply case 1, T(n) = (nlg7)

© 2023 by Jiann-Chyi Rau Algorithms 2-51

Dept. of ECE, TKU
Fall, 2023

Example (III) of Master Theorem

• Maximum-subarray: T(n) = 2T(n/2) + Θ(n)
– a = 2, b = 2, and f(n) = Θ(n)
– nlog a = nlog 2 = n
b 2

– f(n) = (nlog 2)
2

– Apply case 2, T(n) = (nlgn)

© 2023 by Jiann-Chyi Rau Algorithms 2-52