NEW SOUTH WALES
Algorithms:
COMP3121/3821/9101/9801
Aleks Ignjatovic
School of Computer Science and Engineering
University of New South Wales
TOPIC 1: RECURRENCES
COMP3121/3821/9101/9801 1 / 20
Asymptotic notation
Big Oh notation: f (n) = O(g(n)) is an abbreviation for:
There exist positive constants c and n0 such that
0 f (n) c g(n) for all n n0 .
In this case we say that g(n) is an asymptotic upper bound for
f (n).
f (n) = O(g(n)) means that f (n) does not grow substantially faster
than g(n) because a multiple of g(n) eventually dominates f (n).
Clearly, multiplying constants c of interest will be larger than 1,
thus enlarging g(n).
COMP3121/3821/9101/9801 2 / 20
Asymptotic notation
Omega notation: f (n) = (g(n)) is an abbreviation for:
There exists positive constants c and n0 such that
0 c g(n) f (n) for all n n0 .
In this case we say that g(n) is an asymptotic lower bound for
f (n).
f (n) = (g(n)) essentially says that f (n) grows at least as fast as
g(n), because f (n) eventually dominates a multiple of g(n).
Clearly, multiplying constants c of interest will be smaller than 1,
thus shrinking g(n) by a constant factor.
Theta notation: f (n) = (g(n)) iff and only if
f (n) = O(g(n)) and f (n) = (g(n)); thus, f (n) and g(n) have the
same asymptotic growth rate.
COMP3121/3821/9101/9801 3 / 20
Recurrences
Recurrences are important to us because they arise in estimations
of time complexity of divide-and-conquer algorithms.
Merge-Sort(A, p, r) *sorting A[p..r]*
1 if p < r
2 then q b p+r
2 c
3 Merge-Sort(A, p, q)
4 Merge-Sort(A, q + 1, r)
5 Merge(A, p, q, r)
Since Merge(A, p, q, r) runs in linear time, the runtime T (n) of
Merge-Sort(A, p, r) satisfies
n
T (n) = 2T + cn
2
COMP3121/3821/9101/9801 4 / 20
Recurrences
Let a 1 be an integer and b > 1 a real number;
Assume that a divide-and-conquer algorithm:
reduces a problem of size n to a many problems of smaller size n/b;
the overhead cost of splitting up/combining the solutions for size
n/b into a solution for size n is if f (n),
then the time complexity of such algorithm satisfies
n
T (n) = a T + f (n)
b
Note: we should be writing
l n m
T (n) = a T + f (n)
b
but it can be shown that assuming that n is a power of b is OK,
and that the estimate produced is still valid for all n.
COMP3121/3821/9101/9801 5 / 20
n
T (n) = a T + f (n)
b
size of instance = n
a many size of instances = n/b
instances
size of instances = n/b2
. depth of
. recursion:
. log !
. .
. . . .
. .
. . .
. .
. .
size of instances = 1
COMP3121/3821/9101/9801 6 / 20
Some recurrences can be solved explicitly, but this tends to be
tricky.
Fortunately, to estimate efficiency of an algorithm we do not need
the exact solution of a recurrence
We only need to find:
1 the growth rate of the solution i.e., its asymptotic behaviour;
2 the sizes of the constants involved (more about that later)
This is what the Master Theorem provides (when it is
applicable).
COMP3121/3821/9101/9801 7 / 20
Master Theorem:
Let:
a 1 and b > 1 be integers;
f (n) > 0 be a monotonically increasing function;
T (n) be the solution of the recurrence T (n) = a T (n/b) + f (n);
Then:
1 If f (n) = O(nlogb a ) for some > 0, then T (n) = (nlogb a );
2 If f (n) = (nlogb a ), then T (n) = (nlogb a log2 n);
3 If f (n) = (nlogb a+ ) for some > 0, and for some c < 1,
a f (n/b) c f (n)
then T (n) = (f (n));
4 If none of these conditions hold, the Master Theorem is NOT
applicable (in the form presented).
COMP3121/3821/9101/9801 8 / 20
Master Theorem - a remark
Note that for any b > 1,
logb n = logb 2 log2 n;
Since b > 1 is constant (does not depend on n), we have for
c = logb 2 > 0
logb n = c log2 n;
1
log2 n = logb n;
c
Thus,
logb n = (log2 n)
and also
log2 n = (logb n).
So whenever we have f = (g(n) log n) we do not have to specify
what base the log is - all bases produce equivalent asymptotic
estimates.
COMP3121/3821/9101/9801 9 / 20
Master Theorem - Examples
Let T (n) = 4 T (n/2) + n;
then nlogb a = nlog2 4 = n2 ;
thus f (n) = n = O(n2 ) for any < 1.
Condition of case 1 is satisfied; thus, T (n) = (n2 ).
Let T (n) = 2T (n/2) + c n;
then nlogb a = nlog2 2 = n1 = n;
thus f (n) = c n = (n) = (nlog2 2 ).
Thus, condition of case 2 is satisfied; and so,
T (n) = (nlog2 2 log n) = (n log n).
COMP3121/3821/9101/9801 10 / 20
Master Theorem - Examples
Let T (n) = 3 T (n/4) + n;
then nlogb a = nlog4 3 < n0.8 ;
thus f (n) = n = (n0.8+ ) for any < 0.2.
Also, af (n/b) = 3f (n/4) = 3/4 n < c n for c = .8 < 1.
Thus, Case 3 applies, and T (n) = (f (n)) = (n).
Let T (n) = 2 T (n/2) + n log2 n;
then nlogb a = nlog2 2 = n1 = n.
Thus, f (n) = n log2 n = (n).
However, f (n) = n log2 n 6= (n1+ ), no matter how small > 0.
This is because for every > 0, and every c > 0, no matter how
small, log2 n < c n for all sufficiently large n.
Homework: Prove this.
Hint: Use de LHopitals Rule to show that log n/n 0.
Thus, in this case the Master Theorem does not apply!
COMP3121/3821/9101/9801 11 / 20
Master Theorem - Proof:
Since
n
T (n) = a T + f (n) (1)
b
implies (by applying it to n/b in place of n)
n n n
T = aT +f (2)
b b2 b
and (by applying (1) to n/b2 in place of n)
n n n
T = aT +f (3)
b2 b3 b2
and so on . . ., we get
(1)
z }|
{
n n n
T (n) = a T +f (n) = a a T 2
+f + f (n)
b b b
| {z } | {z }
(2) (2)
2 n n
2 n n n
=a T 2
+a f b + f (n) = a aT +f + af b + f (n)
b b3 b2
| {z } | {z }
(3) (3)
3 n
+a2 f bn2 + a f n
=a T 3 b + f (n) = . . .
b
| {z }
COMP3121/3821/9101/9801 12 / 20
Master Theorem Proof:
Continuing in this way logb n 1 many times we get ...
n n n
T (n) = a3 T 3 +a2 f 2 + a f + f (n) =
| {zb } b b
= ...
n n
= ablogb nc T blog nc + ablogb nc1 f blog nc1 + . . .
b b n b n
b
3 n 2
+ a f 3 + a f 2 + af + f (n)
b b b
blogb nc1
logb n
n X n
a T + ai f
blogb n i=0
bi
We now use alogb n = nlogb a :
blogb nc1 n
X
T (n) nlogb a T (1) + ai f (4)
i=0
bi
Note that so far we did not use any assumptions on f (n) . . .
COMP3121/3821/9101/9801 13 / 20
Master Theorem Proof:
Case 1: f (m) = O(mlogb a )
blogb nc1 n blogb nc1 n logb a
X X
ai f = ai O
i=0
bi i=0
bi
blogb nc1 blogb nc1
ai
X n logb a X
i logb a
= O a = O n
i=0
bi i=0
(bi )logb a
blogb nc1 blogb nc1
logb a
X a i
logb a
X a i
= O n = O n
blogb a
i=0 i=0
blogb a b
blogb nc1 i
blogb nc1
log a
X a b log a
X i
= O n b = O n b (b )
i=0
a i=0
! m
blogb nc
logb a (b ) 1 X q m+1 1
=O n ; we are using qm =
b 1
i=0
q1
COMP3121/3821/9101/9801 14 / 20
Master Theorem Proof:
Case 1 - continued:
blogb nc1
!
blogb nc
X n
logb a (b ) 1
ai f i = O n
i=0
b b 1
bblogb nc 1
logb a
= O n
b 1
n 1
= O nlogb a
b 1
log a
n b nlogb a
=O
b 1
= O nlogb a
blogb nc1 n
X
Since we had: T (n) nlogb a T (1) + ai f we get:
i=0
bi
T (n) nlogb a T (1) + O nlogb a
= nlogb a
COMP3121/3821/9101/9801 15 / 20
Master Theorem Proof:
Case 2: f (m) = (mlogb a )
blogb nc1 n blogb nc1 n logb a
X X
ai f = ai
i=0
bi i=0
bi
blogb nc1 n logb a
X i
= a
i=0
bi
blogb nc1 i
X a
= nlogb a
i=0
(bi )logb a
blogb nc1
X a i
= nlogb a
i=0
blogb a
blogb nc1
X
= nlogb a 1
i=0
= nlogb a blogb nc
COMP3121/3821/9101/9801 16 / 20
Master Theorem Proof:
Case 2 (continued):
Thus,
blogb nc1 n
X
ai f = nlogb a logb n = nlogb a log2 n
i=0
bi
because logb n = log2 n logb 2 = (log2 n). Since we had (1):
blogb nc1 n
X
T (n) nlogb a T (1) + ai f
i=0
bi
we get:
T (n) nlogb a T (1) + nlogb a log2 n
= nlogb a log2 n
COMP3121/3821/9101/9801 17 / 20
Master Theorem Proof:
Case 3: f (m) = (mlogb a+ ) and a f (n/b) c f (n) for some 0 < c < 1.
c
We get by substitution: f (n/b) f (n)
a
c
f (n/b2 ) f (n/b)
a
c
f (n/b3 ) f (n/b2 )
a
...
c
f (n/bi ) f (n/bi1 )
a
By chaining these inequalities we get
c c c c2
f (n/b2 ) f (n/b) f (n) = 2 f (n)
a | {z } a |a {z } a
c c c2 c3
f (n/b3 ) f (n/b2 ) 2 f (n) = 3 f (n)
a | {z } a |a {z } a
...
c c ci1 ci
f (n/bi ) f (n/bi1 ) i1 f (n) = i f (n)
a | {z } a |a {z } a
COMP3121/3821/9101/9801 18 / 20
Master Theorem Proof:
Case 3 (continued):
ci
We got f (n/bi ) f (n)
ai
Thus,
blogb nc1 blogb nc1
X n X ci X f (n)
ai f ai f (n) < f (n) ci =
i=0
bi i=0
ai i=0
1 c
Since we had (1):
blogb nc1 n
X
T (n) nlogb a T (1) + ai f
i=0
bi
and since f (n) = (nlogb a+ ) we get:
T (n) < nlogb a T (1) + O (f (n)) = O (f (n))
but we also have
T (n) = aT (n/b) + f (n) > f (n)
thus,
T (n) = (f (n))
COMP3121/3821/9101/9801 19 / 20
Master Theorem Proof: Homework
Exercise 1: Show that condition
f (n) = (nlogb a+ )
follows from the condition
a f (n/b) c f (n) for some 0 < c < 1.
Exercise 2: Estimate T (n) for
T (n) = 2 T (n/2) + n log n
Note: we have seen that the Master Theorem does NOT apply, but the technique
used in its proof still works! Just unwind the recurrence and sum up the logarithmic
overheads.
COMP3121/3821/9101/9801 20 / 20
