Elementary Algorithmic

 Looping
Outline
 Analysis of Algorithm
 The efficient algorithm
 Average, Best and Worst case analysis
 Asymptotic Notations
 Analyzing control statement
 Loop invariant and the correctness of the algorithm
 Sorting Algorithms and analysis: Bubble sort,
Selection sort, Insertion sort, Shell sort and Heap
sort
 Sorting in linear time : Bucket sort, Radix sort and
Counting sort
 Amortized analysis
Introduction

What is Analysis of an Algorithm?

 Analyzing an algorithm means calculating/predicting the
resources that the algorithm requires.
 Analysis provides theoretical estimation for the required
resources of an algorithm to solve a specific computational
problem.
 Two most important resources are computing time (time
complexity) and storage space (space complexity).
Why Analysis is required?
 By analyzing some of the candidate algorithms for a problem,
the most efficient one can be easily identified.

3
Efficiency of Algorithm

consumed in solving a problem of size 𝑛.

 The efficiency of an algorithm is a measure of the amount of resources

 An algorithm must be analyzed to determine its resource usage.

 Two major computational resources are execution time and memory
space.
 Memory Space requirement can not be compared directly, so the
important resource is computational time required by an algorithm.
 To measure the efficiency of an algorithm requires to measure its
execution time using any of the following approaches:
1. Empirical Approach: To run it and measure how much processor time is needed.
2. Theoretical Approach: Mathematically computing how much time is needed as a
function of input size.

4
How Analysis is Done?
Empirical (posteriori)
Theoretical (priori) approach
approach
 Programming different  Determining
competing techniques & mathematically the
running them on various resources needed by each
inputs using computer. algorithm.
 Implementation of different  Uses the algorithm instead
techniques may be difficult. of an implementation.
 The same hardware and  The speed of an algorithm
software environments can be determined
must be used for comparing independent of the
two algorithms. hardware/software
 Results may not be environment.
indicative of the running  Characterizes running time
time on other inputs not as a function of the input
included in the experiment. size considers all possible
5
Time Complexity
 Time complexity of an algorithm quantifies the amount of time taken by
an algorithm to run as a function of the length of the input.
 Running time of an algorithm depends upon,
1. Input Size
2. Nature of Input
 Generally time grows with the size of input, for example, sorting 100
numbers will take less time than sorting of 10,000 numbers.
 So, running time of an algorithm is usually measured as a function of input
size.
 Instead of measuring actual time required in executing each statement in
the code, we consider how many times each statement is executed.
 So, in theoretical computation of time complexity, running time is
measured in terms of number of steps/primitive operations performed.

6
Linear Search
 Suppose, you are given a jar containing some business cards.
 You are asked to determine whether the name “Bill Gates" is in the jar.
 To do this, you decide to simply go through all the cards one by one.
 How long this takes?
 Can be determined by how many cards are in the jar, i.e., Size of Input.
 Linear search is a method for finding a particular value from the given list.
 The algorithm checks each element, one at a time and in sequence, until
the desired element is found.
 Linear search is the simplest search algorithm.
 It is a special case of brute-force search.

8
Linear Search – Example

Search for in given array

Comparing value of ith index with the given element one by one, until
we get the required element or end of the array
Step 1: i=1 Step 3: i=3

i i
𝟐≠𝟏 𝟑≠𝟏
Step 2: i=2 Step 4: i=4

𝟏
i i
𝟗≠𝟏 Element found at ith index, i=4
9
Linear Search - Algorithm
# Input : Array A, element x
# Output : First index of element x in A or -1 if not found

Algorithm: Linear_Search
for i = 1 to last index of A
if A[i] equals element x
return i
return -1

10
Linear Search - Analysis
 The required element in the given array can be found at,

Case 1: element 2 Case 2: element 3 Case 3: element 7 at

which is at the first anywhere after the last position or
position so minimum first position so, an element does not
comparison is required average number of found at all, maximum
comparison is required comparison is required
Best Case Average Case Worst Case
Worst
Case

𝟑
Search 𝟕
for

Best
Case Average
Case
12
Analysis of Algorithm

Best Case Average Case Worst Case

Resource usage is Resource usage is Resource usage is
minimum average maximum
Algorithm’s behavior Algorithm’s behavior Algorithm’s behavior
under optimal under random under the worst
condition
Minimum number of condition
Average number of condition
Maximum number of
steps or operations steps or operations steps or operations
Lower bound on Average bound on Upper bound on
running time running time running time
Generally do not Average and worst-case performances are the
occur in real most used in algorithm analysis.

13
Book Finder Example  Suppose, you are writing a program
to find a book from the shelf.
 For any required book, it will start
checking books one by one from the
bottom.
 If you wanted Harry Potter 3, it
would only take 3 actions (or tries)
because it’s the third book in the
sequence.
 If Harry Potter 7 — it’s the last book
so it would have to check all 7
books.
 What if there are total 10 books?
How about 10,00,000 books? It
would take 1 million tries.

14
Number Sorting - Example
 Suppose you are sorting numbers in Ascending / Increasing order.
 The initial arrangement of given numbers can be in any of the following
three orders.
1. Numbers are already in required order, i.e., Ascending order
No change is required – Best Case
2. Numbers are randomly arranged initially.
Some numbers will change their position – Average Case
3. Numbers are initially arranged in Descending or Decreasing order.
All numbers will change their position – Worst Case

15
Number Sorting - Example
 Suppose you are sorting numbers in Ascending / Increasing order.
 The initial arrangement of given numbers can be in any of the following
three orders.
Case 1: Numbers are Case 2: Numbers are Case 3: Numbers are
already in required randomly arranged initially arranged in
order, i.e., Ascending initially. Some numbers Descending order so,
order will change their all numbers will
No change is required position change their position
Best Case Average Case Worst Case

16
Best, Average, & Worst Case

Problem Best Case Average Case Worst Case

Linear Element at the Element in any of Element at last
Search first position the middle position or not
Book Finder The first book positions
Any book in- present
The last book
between
Sorting Already sorted Randomly Sorted in reverse
arranged order

17
What is a Good Algorithm?
 Efficient
 Running time
 Space used
 Efficiency as a function of input size
 The number of bits in an input number
 Number of data elements(Numbers and Points)

18
Measuring the Running Time
 How should we measure the running time of an algorithm?
 Experimental Study
 Write a program that implements the algorithm
 Run the program with data sets of varying size and composition.
 Use a method like System.currentTimeMillis() to get an accurate measure of the
actual running time

19
Limitations of Experimental Studies
 It is necessary to implement and test the algorithm in order to determine
its running time.
 Experiments can be done only on a limited set of inputs, and may not be
indicative of the running time on other inputs not included in the
experiment.
 In order to compare two algorithms, the same hardware and software
environments should be used.

20
Best/Worst/Average Case
 For a specific size of input n, investigate running times for different input
instances:

21
Best/Worst/Average Case
 For inputs of all sizes:

22
Asymptotic Notations
Given two algorithms A1 and A2 for a problem, how do we decide
 which one runs faster?
 What we need is a platform independent way of comparing algorithms.
 Solution: Count the worst-case number of basic operations b(n) for inputs
of size n and then analyse how this function b(n) behaves as n grows. This
is known as worst-case analysis.
 Observations regarding worst-case analysis:
 Usually, the running time grows with the input size n.
 Consider two algorithm A1 and A2 for the same problem. A1 has a worst-case
running time (100n + 1) and A2 has a worst-case running time (2n2 + 3n + 1). Which
one is better?
 A2 runs faster for small inputs (e.g., n = 1, 2)
 A1 runs faster for all large inputs (for all n ≥ 49)
 We would like to make a statement independent of the input size.
 Solution: Asymptotic analysis
 We consider the running time for large inputs.
 A1 is considered better than A2 since A1 will beat A2 eventually

24
continue..
Solution: Do an asymptotic worst-case analysis.
 Observations regarding asymptotic worst-case analysis:
 It is difficult to count the number of operations at an extremely fine level
and keep track of these constants.
 Asymptotic analysis means that we are interested only in the rate of
growth of the running time function w.r.t. the input size. For example, note
that the rates of growth of functions (n2 + 5n + 1) and (n2 + 2n + 5) is
determined by the n2 (quadratic) term. The lower order terms are
insignificant. So, we may as well drop them.
 The nature of growth rate of functions 2n2 and 5n2 are the same. Both are
quadratic functions. It makes sense to drop these constants too when one
is interested in the nature of the growth functions.
 These constants typically depends upon system you are using, such as
hardware, compiler etc.
 We need a notation to capture the above ideas.
25
Introduction
 The theoretical (priori) approach of analyzing an algorithm to measure the
efficiency does not depend on the implementation of the algorithm.
 In this approach, the running time of an algorithm is describes as
Asymptotic Notations.
 Computing the running time of algorithm’s operations in mathematical
units of computation and defining the mathematical formula of its run-
time performance is referred to as Asymptotic Analysis.
 An algorithm may not have the same performance for different types of
inputs. With the increase in the input size, the performance will change.
 Asymptotic analysis accomplishes the study of change in performance of
the algorithm with the change in the order of the input size.
 Using Asymptotic analysis, we can very well define the best case, average
case, and worst case scenario of an algorithm.

26
Asymptotic Notations
 Asymptotic notations are mathematical notations used to represent the
time complexity of algorithms for Asymptotic analysis.
 Following are the commonly used asymptotic notations to calculate the
running time complexity of an algorithm.
1. Notation
2. Notation
3. Notation
 This is also known as an algorithm’s growth rate.
 Asymptotic Notations are used,
1. To characterize the complexity of an algorithm.
2. To compare the performance of two or more algorithms solving the same problem.

27
1. -Notation (Big notation) (Upper Bound)
 The notation is the formal way to express the upper bound of an
algorithm's running time.
 It measures the worst case time complexity or the longest amount of time
an algorithm can possibly take to complete.
 For a given function , we denote by the set of functions,

= { : there exist positive constants and such that for all }

28
Big() Notation  is an asymptotically upper bound
for .

 implies:
𝒄 . 𝒈(𝒏)

 For any value of n, the running time

𝒇 (𝒏)
of an algorithm does not cross the
time provided by O(g(n)).

 Time taken by a known algorithm to

solve a problem with worse case
input gives the upper bound.
𝒏 𝟎 𝒇 (𝒏)=𝑶 (𝒈 (𝒏)) 𝒏

29
Example
For a function f(n) and g(n) there are positive constants c and n 0 such that :
f(n) ≤ c.g(n) for n ≥ n0

Conclusion: 2n+6 is O(n)

30
Example
On the other hand n2 is not O(n) because
there is no c and n0 such that:
n2 ≤ cn for n ≥ n0

The graph shows that no matter how large

a c is chosen there is an n big enough
that n2 > cn.

31
Simple rule
 Drop lower order term and constant factor

 50 n log n is O(n log n)

 7n -3 is O(n)
 8n2 log n + 5n2 + n is O(n2 log n )

 Use O-notations to express number of primitive operations executed as

function of input size.
 Comparing asymptotic running time:
 an algorithm that runs O(n) times is better than one that runs in O(n2)
times
 similarly, O(logn) is better than O(n)
 Hierarchy of function: log n < n < n2 < n3 < 2n

32
2. -Notation (Omega notation) (Lower Bound)
 Big Omega notation () is used to define the lower bound of any algorithm
or we can say the best case of any algorithm.
 This always indicates the minimum time required for any algorithm for all
input values, therefore the best case of any algorithm.
 When a time complexity for any algorithm is represented in the form of
big-Ω, it means that the algorithm will take at least this much time to
complete it's execution. It can definitely take more time than this too.
 For a given function 𝑔(𝑛), we denote by 𝑔(𝑛)) the set of functions,

there exist positive constants and such that for all

33
Big() Notation  is an asymptotically lower bound
for

𝒇 (𝒏)  implies:

𝒄 . 𝒈(𝒏)  if there exists a positive constant c

such that it lies above cg(n), for
sufficiently large n.

 For any value of n, the minimum

time required by the algorithm is
given by Omega Ω(g(n)).
𝒏 𝟎 𝒇 (𝒏)=𝜴 (𝒈 (𝒏)) 𝒏

34
3. -Notation (Theta notation) (Same order)
 The notation is the formal way to enclose both the lower bound and the
upper bound of an algorithm's running time.
 Since it represents the upper and the lower bound of the running time of
an algorithm, it is used for analyzing the average case complexity of an
algorithm.
 The time complexity represented by the Big- notation is the range within
which the actual running time of the algorithm will be.
 So, it defines the exact Asymptotic behavior of an algorithm.
 For a given function 𝑔(𝑛), we denote by θ(𝑔(𝑛)) the set of functions,
there exist positive constants

35
-Notation  is a set, we can write to indicate
𝒄 𝟐 .𝒈(𝒏) that is a member of

 is an asymptotically tight bound for

𝒇 (𝒏)
 implies:
𝒄 𝟏 .𝒈(𝒏)
If a function f(n) lies anywhere in
between c1g(n) and c2g(n) for all n ≥
n0, then f(n) is said to be
asymptotically tight bound.
𝒏𝟎
𝒏
𝒇 (𝒏)=𝜽(𝒈 (𝒏)) f(n) is Θ(g(n)) if and only if f(n) is
Ο(g(n)) and f(n) is Ω(g(n))

36
Asymptotic Notations
1. O-Notation (Big O notation) (Upper Bound)

= { : there exist positive constants and such thatfor

all}
𝐟 (𝐧 )=𝐎 ( 𝐠 ( 𝐧 ))

2. Ω-Notation (Omega notation) (Lower Bound)

= { : there exist positive constants and such that for

all }
𝐟 ( 𝐧 ) =Ω(𝐠 ( 𝐧 ))

3. θ-Notation (Theta notation) (Same order)

= { : there exist positive constants , andsuch that for

all }
𝐟 (𝐧 )=𝛉 (𝐠 ( 𝐧))

37
Asymptotic Notations – Examples
 Example 1:  Example 2:
and and
Algo. 1 Algo. 2 Algo. 1 Algo. 2
running running running running
time time time time

𝑓 ( 𝑛 ) ≥ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =Ω(𝑔 (𝑛)) 𝑓 ( 𝑛 ) ≤ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =O(𝑔 (𝑛))

1 1 1 1 1 1
2 4 2 2 2 4
3 9 3 3 3 9
4 1 4 4 4 1
5 6
2 5 5 5 6
2
5 5
38
Asymptotic Notations – Examples
 Example 3: and

𝑓 ( 𝑛 ) ≤ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =O(𝑔 (𝑛))

1 1 2
2 4 4
3 9 8
4 1 1
6 6 Here for ,
5 2 3
5 2
6 3 6
6 4
7 4 12
9 8

39
Asymptotic Notations –  Example 4:
is in the order of , or
Examples is order , or

𝒇 (𝒏)=𝑶 (𝒈(𝒏))
g (n)=n +1
2
Value of function 

f(n)=30n+8
 In general, any function is faster-
growing than any function.

Base value
Increasing n 

40
Common Orders of Magnitude

4 2 8 16 64 16 24
16 4 64 256 4096 65536 2.09 x
1013
64 6 384 4096 262144 1.84 × 1.26 x
1019 1029
256 8 2048 65536 16777216 1.15 ×
1077
102 10 10240 1048576 1.07 × 109 1.79 ×
4 10308
409 12 49152 1677721 6.87 × 101233
6 6 1010 41
Comparison
 O(c) < O(log logn) < O(logn) < O(n1/2) < O(n) < O(nlogn) < O(n2) < O(n3)
< O(nk) < O(2n) < O(nn) < O(22^n)

42
Asymptotic Notations in Equations
 Consider an example of buying elephants and goldfish:
Cost = cost_of_elephants + cost_of_goldfish
Negligib
Cost ≈ cost_of_elephants (approximation) le

𝑂( 𝑓(𝑛)
 Maximum Rule: Let, the max rule says that:

+𝑔(𝑛))=𝑂(max⁡(𝑓(𝑛),𝑔(𝑛)))

1. - is

 The low order terms in a function are relatively insignificant for large 𝒏

44
Exercises
1. Express the function in terms of θ notation.
2. Express in terms of O notation.
3. Express in terms of O notation.
4. Prove or disprove (i) Is 2n+1 = O(2n) (ii) Is 22n = O(2n)
5. Check the correctness for the following equality,
6. Find θ notation for the following function

7. Find O notation for the following function

8. Find Ω notation for the following function

45
Methods of proving Asymptotic Notations
1) Proof by definition : In this method, we apply the formal definition of
the asymptotic notation, and find out the values
of constants c > 0 and n0 > 0, such that the required notation is proved.

2) Proof by Limit Rules : In this method, we apply certain rules of limit,

and then prove the required notation.

46
Proof by definition
 Prove the following statements :
1. n2 + n = O(n ) 3)
≈ O(n
2

According to the formal definition, let f(n) = n2 + n and g(n) = n2

Find the values of constants c > 0 and n0 > 0, such that 0 ≤ f(n) ≤ c g(n), for all n ≥ n 0
(condition for Big-O notation)2
≈ Ω(n)
2. n3 + 4n2 = Ω(n2
)
According to the formal definition, let f(n) = n3 + 4n2 and g(n) = n2
Find the values of constants c > 0 and n0 > 0, such that 0 ≤ c g(n) ≤ f(n), for all n ≥ n 0
(condition for Big-Ω notation)
3. n2 + n = Θ(n2)
According to the formal definition, let f(n) = n2 + n and g(n) = n2
Find the values of constants c1 > 0, c2 > 0, and n0 > 0, such that 0 ≤ c1 g(n) ≤ f(n)
≤ c2 g(n), for all n ≥ n0 (condition for Θ notation)

47
Proof by Limit Rules
If f(n) and g(n) are asymptotically increasing functions, then the following
rules hold true:

Prove that : √n grows asymptotically faster than log n.

Proof: Let us consider f(n) = √n and g(n) = log n

We compute and then based on the result, the

specific

“Limit Rule” proves the desired result. 48

Little-Oh and Little-Omega
f(n)=o(g(n)) => For every c, there should exist n0 , s.t. f(n) ≤ c g(n) for n ≥
n0
f(n)=ω(g(n)) => For every c, there should exist n0 , s.t. f(n) ≥ c g(n) for n ≥
n0

Analogy with real numbers

o f(n) = O(g(n)) ≅ f ≤ g
o f(n) = Ω(g(n)) ≅ f ≥ g
o f(n) = Θ(g(n)) ≅ f = g
o f(n) = o(g(n)) ≅ f < g
o f(n) = ω(g(n)) ≅ f > g

49
Math You Need to Review
Properties of logarithms:

Properties of exponentials:

Geometric progression:

Arithmetic progression:

50
Examples
 Example: Find upper bound of running time of constant function f(n) =
6993.
 To find the upper bound of f(n), we have to find c and n0 such that 0 ≤ f (n) ≤ c.g(n) for
all n ≥ n0
 0 ≤ f (n) ≤ c × g (n)
 0 ≤ 6993 ≤ c × g (n)
 0 ≤ 6993 ≤ 6993 x 1
 So, c = 6993 and g(n) = 1
 Any value of c which is greater than 6993, satisfies the above inequalities, so all such
values of c are possible.
 0 ≤ 6993 ≤ 8000 x 1 → true
 0 ≤ 6993 ≤ 10500 x 1 → true
 Function f(n) is constant, so it does not depend on problem size n. So n0= 1
 f(n) = O(g(n)) = O(1) for c = 6993, n0 = 1
 f(n) = O(g(n)) = O(1) for c = 8000, n0 = 1 and so on.
51
Find upper bound of running time of a linear function f(n) = 6n + 3.
To find upper bound of f(n), we have to find c and n0 such that 0 ≤ f (n) ≤ c × g (n) for all n ≥ n0
0 ≤ f (n) ≤ c × g (n)
0 ≤ 6n + 3 ≤ c × g (n)
0 ≤ 6n + 3 ≤ 6n + 3n, for all n ≥ 1 (There can be such infinite possibilities)
0 ≤ 6n + 3 ≤ 9n
So, c = 9 and g (n) = n, n0 = 1

Tabular Approach
0 ≤ 6n + 3 ≤ c × g (n)
0 ≤ 6n + 3 ≤ 7 n
Now, manually find out the proper n0, such that f
(n) ≤ c.g (n)

From Table, for n ≥ 3, f (n) ≤ c × g (n) holds true.

So, c = 7, g(n) = n and n0 = 3, There can be such
multiple pair of (c, n0)

f(n) = O(g(n)) = O(n) for c = 9, n0 = 1 52

 Example: Find lower bound of running time of quadratic function f(n) = 3n 2
+ 2n + 4.
 To find lower bound of f(n), we have to find c and n0 such that 0 ≤ c.g(n) ≤ f(n) for all n
³ n0
 0 ≤ c × g(n) ≤ f(n)
 0 ≤ c × g(n) ≤ 3n2 + 2n + 4
 0 ≤ 3n2 ≤ 3n2 + 2n + 4, → true, for all n ≥ 1
 0 ≤ n2 ≤ 3n2 + 2n + 4, → true, for all n ≥ 1
 Above both inequalities are true and there exists such infinite inequalities.
 So, f(n) = Ω (g(n)) = Ω (n2) for c = 3, n0 = 1
 f(n) = Ω (g(n)) = Ω (n2) for c = 1, n0 = 1

53
 Find tight bound of running time of a cubic function f(n) = 2n 3 + 4n + 5.
 To find tight bound of f(n), we have to find c1, c2 and n0 such that, 0 ≤ c1 × g(n) ≤
f(n) ≤ c2 × g(n) for all n ≥ n0
 0 ≤ c1 × g(n) ≤ 2n3 + 4n + 5 ≤ c2 × g(n)
 0 ≤ 2n3 ≤ 2n3 + 4n + 5 ≤ 11n3, for all n ≥ 1
 Above inequality is true and there exists such infinite inequalities. So,
 f(n) = Θ(g(n)) = Θ(n3) for c1 = 2, c2 = 11, n0 = 1

54
General Problems
 Example: Show that : (i) 3n + 2 = Θ(n) (ii) 6*2n + n2 = Θ(2n)
 (i) 3n + 2 = Θ(n)
 To prove above statement, we have to find c1, c2 and n0 such that, 0 ≤
c1× g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
 0 ≤ c1× g(n) ≤ 3n + 2 ≤ c2 × g(n)
 0 ≤ 2n ≤ 3n + 2 ≤ 5n, for all n ≥ 1
 So, f(n) = Θ(g(n)) = Θ(n) for c1 = 2, c2 = 5 n0 = 1
 (ii) 6*2n + n2 = Θ(2n)
 To prove above statement, we have to find c1, c2 and n0 such that, 0 ≤
c1× g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
 0 ≤ c1× g(n) ≤ 6*2n + n2 ≤ c2 × g(n)
 0 ≤ 6.2n ≤ 6*2n + n2 ≤ 7*2n, for all n ≥ 1
 So, f(n) = Θ(g(n)) = Θ(2n) for c1 = 6, c2 = 7 n0 = 1

55
Thank You!