ITMC103

UNIT1-Algorithms &Complexity
Analysis
Instructor: Neha Sharma
Algorithms
• Algorithm is ‘a formally defined procedure for performing some
calculation
• If a procedure is formally defined, then it can be implemented
using a formal language, and such a language is known as a
programming language
• Algorithms are mainly used to achieve software reuse.
• Once we have an idea or a blueprint of a solution, we can
implement it in any high-level language like C, C++, or Java

2
Algorithms
• Algorithms are used to manipulate the data contained in data structures.
• When working with data structures, algorithms are used to perform
operations on the stored data.
• A complex algorithm is often divided into smaller units called modules. This
process of dividing an algorithm into modules is called modularization.
• The key advantages of modularization are :
• It makes the complex algorithm simpler to design and implement.
• Each module can be designed independently.
• While designing one module, the details of other modules can be ignored,
thereby enhancing clarity in design which in turn simplifies
implementation, debugging, testing, documenting, and maintenance of the
overall algorithm.
Algorithms
• There are two main approaches to design an algorithm—top-down
approach and bottom-up approach
• A top-down design approach starts by dividing the complex algorithm
into one or more modules.
• A bottom-up approach is just the reverse of top-down approach.
• In the bottom-up design, we start with designing the most basic or
concrete modules and then proceed towards designing higher level
modules.
Control Structures in Algorithms
• An algorithm may employ one of the following control structures:
• Sequence-each step of an algorithm is executed in a specified order.
• Decision-when the execution of a process depends on the outcome
of some condition e.g
IF ‘condition’ Then
process1
ELSE
process2
Control Structures in Algorithms
• Repetition-involves executing one or more steps for a number of times,
can be implemented using constructs such as while, do–while, and for
loops
• Eg.

• Exercise:
Write an algorithm to find the larger of two numbers.
Write an algorithm to find the sum of first N natural numbers
Good Algorithms?
• Runs in less time

• Consumes less memory

But computational resources (time complexity) is usually more

important

7
Measuring Efficiency
• The efficiency of an algorithm is a measure of the amount of resources
consumed in solving a problem of size n.

• The resource we are most interested in is time

• We can use the same techniques to analyze the consumption of
other resources, such as memory space.

• It would seem that the most obvious way to measure the efficiency of
an algorithm is to run it and measure how much processor time is
needed?

8
Factors
 Hardware

 Operating System

 Compiler

 Size of input

 Nature of Input

 Algorithm Which should be improved?

RUNNING TIME OF AN ALGORITHM
• Depends upon
• Input Size
• Nature of Input

• Generally time grows with size of input, so running time of an algorithm is

usually measured as function of input size.

• Running time is measured in terms of number of steps/primitive

operations performed

• Independent from machine, OS

10
Finding running time of an Algorithm / Analyzing an
Algorithm

• Running time is measured by number of steps/primitive

operations performed

• Steps mean elementary operations like

• ,+, *,<, =, A[i] etc

• We will measure number of steps taken in term of size of

input
11
 Simple Example (1)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}

How should we analyse this?

12
 Simple Example (2)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 1,2,8: Once
return s;
6 7
3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

13
 Simple Example (3)
Estimated running time for different values of N:
Growth of 5n+3
N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear

proportion to N for this function “Sum”

14
What Dominates in Previous Example?

What about the +3 and 5 in 5N+3?

• As N gets large, the +3 becomes insignificant
• 5 is inaccurate, as different operations require varying amounts of time and also does not have
any significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on the

highest order term:
• Drop lower order terms such as +3
• Drop the constant coefficient of the highest order term i.e. N

15
Asymptotic Complexity

• The 5N+3 time bound is said to "grow asymptotically" like N

• This gives us an approximation of the complexity of the algorithm

• Ignores lots of (machine dependent) details, concentrate on the bigger

picture

16
Asymptotic analysis
• Asymptotic analysis of an algorithm refers to defining the
mathematical boundation/framing of its run-time performance
• Asymptotic analysis is input bound i.e., if there's no input to the
algorithm, it is concluded to work in a constant time. Other than the
"input" all other factors are considered constant.
• Asymptotic analysis refers to computing the running time of any
operation in mathematical units of computation.
Asymptotic analysis
• For example, the running time of one operation is computed as f(n)
and may be for another operation it is computed as g(n 2 )
• This means the first operation running time will increase linearly with
the increase in n and the running time of the second operation will
increase exponentially when n increases.
• Similarly, the running time of both operations will be nearly the same
if n is significantly small.
Asymptotic analysis
• The time required by an algorithm falls under three types −
• Best Case − Minimum time required for program execution.
• Average Case − Average time required for program execution.
• Worst Case − Maximum time required for program execution.
COMPARING FUNCTIONS: ASYMPTOTIC NOTATION

• Big Oh Notation: Upper bound

• Omega Notation: Lower bound

• Theta Notation: Tighter bound

20
 Big Oh Notation [1]

If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) is order g(N)", or "f(N) is big-O of g(N)")

if there are constants c and N0 such that for N > N0,
f(N) ≤ c * g(N)
for all sufficiently large N.
21
 Big Oh Notation [2]

The notation Ο(n) is the formal

way to express the upper
bound of an algorithm's
running time.

It measures the worst case

time complexity or the longest
amount of time an algorithm
can possibly take to complete.

22
O(f(n))

23
 Example (2): Comparing Functions

• Which function is better? 4000

10 n2 Vs n3 3500

3000

2500

10 n^2
2000
n^3

1500

1000

500

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

24
 Comparing Functions

• As inputs get larger, any algorithm of a smaller

order will be more efficient than an algorithm of
a larger order
0.05 N2 = O(N2)
Time (steps)

3N = O(N)

N = 60 Input (size)

25
 Big-Oh Notation

• Even though it is correct to say “7n - 3 is O(n3)”, a

better statement is “7n - 3 is O(n)”, that is, one should
make the approximation as tight as possible

• Simple Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)

26
 Big Omega Notation
• If we wanted to say “running time is at least…” we use Ω

• Big Omega notation, Ω, is used to express the lower bounds on

a function.

• If f(n) and g(n) are two complexity functions then we can say:

f(n) is Ω(g(n)) if there exist positive numbers c and n 0 such that 0<=f(n)>=cΩ (n) for all n>=n 0

27
 Big Theta Notation

• If we wish to express tight bounds we use the theta

notation, Θ

• f(n) = Θ(g(n)) means that f(n) = O(g(n)) and f(n) = Ω(g(n))

28
 What does this all mean?

• If f(n) = Θ(g(n)) we say that f(n) and g(n) grow at the same rate,
asymptotically

• If f(n) = O(g(n)) and f(n) ≠ Ω(g(n)), then we say that f(n) is

asymptotically slower growing than g(n).

• If f(n) = Ω(g(n)) and f(n) ≠ O(g(n)), then we say that f(n) is

asymptotically faster growing than g(n).

29
 Which Notation do we use?
• To express the efficiency of our algorithms which of the three
notations should we use?

• As computer scientist we generally like to express our

algorithms as big O since we would like to know the upper
bounds of our algorithms.

• Why?

• If we know the worse case then we can aim to improve it and/or

avoid it.
30
 Performance Classification
f(n) Classification
1 Constant: run time is fixed, and does not depend upon n. Most instructions are executed once, or
only a few times, regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which
solve large problems by transforming them into smaller problems. Exp : binary Search

n Linear: run time varies directly with n. Typically, a small amount of processing is done on each
element. Exp: Linear Search

n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem
down into smaller sub-problems, solves them independently, then combines solutions. Exp: Merge

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically
the program processes all pairs of input (e.g. in a double nested loop). Exp: Insertion Search

n3 Cubic: when n doubles, runtime increases eightfold. Exp: Matrix

2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force”
solution. Exp: Brute Force.
Note: logn, n, nlogn, n2>> less Input>>Polynomial
n3, 2n>>high input>> non polynomial
31
 Size does matter[1]

What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~10 76

32
 Complexity Classes

Time (steps)

33
 Size does matter[2]

• Suppose a program has run time O(n!) and the run time
for
n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours

For n = 16, the run time is 2 months

For n = 18, the run time is 50 years

For n = 20, the run time is 200 centuries

34
 Standard Analysis Techniques

• Constant time statements

• Analyzing Loops

• Analyzing Nested Loops

• Analyzing Sequence of Statements

• Analyzing Conditional Statements

35
 Constant time statements

• Simplest case: O(1) time statements

• Assignment statements of simple data types

int x = y;

• Arithmetic operations:
x = 5 * y + 4 - z;

• Array referencing:
A[j] = 5;

• Array assignment:
 j, A[j] = 5;

• Most conditional tests:

if (x < 12) ...
36
 Analyzing Loops[1]
• Any loop has two parts:
• How many iterations are performed?
• How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;

• Loop executes N times (0..N-1)

• 4 = O(1) steps per iteration

• Total time is N * O(1) = O(N*1) = O(N)

37
 Analyzing Loops[2]
• What about this for loop?
int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;

• Loop executes 100 times

• 4 = O(1) steps per iteration

• Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

38
 Analyzing Loops – Linear Loops

• Example (have a look at this code segment):

• Efficiency is proportional to the number of iterations.

• Efficiency time function is :
f(n) = 1 + (n-1) + c*(n-1) +( n-1)
= (c+2)*(n-1) + 1
= (c+2)n – (c+2) +1
• Asymptotically, efficiency is : O(n)

39
 Analyzing Nested Loops[1]
• Treat just like a single loop and evaluate each level of
nesting as needed:

int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

• Start with outer loop:

• How many iterations? N
• How much time per iteration? Need to evaluate inner loop

• Inner loop uses O(N) time

• Total time is N * O(N) = O(N*N) = O(N 2)

40
 Analyzing Nested Loops[2]
• What if the number of iterations of one loop depends on
the counter of the other?

int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

• Analyze inner and outer loop together:

• Number of iterations of the inner loop is:

• 0 + 1 + 2 + ... + (N-1) = O(N2)
41
 How Did We Get This Answer?

• When doing Big-O analysis, we sometimes have to

compute a series like: 1 + 2 + 3 + ... + (n-1) + n

• i.e. Sum of first n numbers. What is the complexity of

this?

• Gauss figured out that the sum of the first n numbers is

always:

42
 Sequence of Statements
• For a sequence of statements, compute their complexity
functions individually and add them up

• Total cost is O(n2) + O(n) +O(1) = O(n2)

43
 Conditional Statements
• What about conditional statements such as

if (condition)
statement1;
else
statement2;

• where statement1 runs in O(n) time and statement2 runs in O(n 2) time?

• We use "worst case" complexity: among all inputs of size n, what is the
maximum running time?

• The analysis for the example above is O(n2)

44
 Deriving A Recurrence Equation
• So far, all algorithms that we have been analyzing have been non
recursive

• Example : Recursive power method

• If N = 1, then running time T(N) is 2

• However if N ≥ 2, then running time T(N) is the cost of each step taken plus time required to
compute power(x,n-1). (i.e. T(N) = 2+T(N-1) for N ≥ 2)

• How do we solve this? One way is to use the 45

iteration method.
 Iteration Method

• This is sometimes known as “Back Substituting”.

• Involves expanding the recurrence in order to see a pattern.

• Solving formula from previous example using the iteration

method :
• Solution : Expand and apply to itself :
Let T(1) = n0 = 2
T(N) = 2 + T(N-1)
= 2 + 2 + T(N-2)
= 2 + 2 + 2 + T(N-3)
= 2 + 2 + 2 + ……+ 2 + T(1)
= 2N + 2 remember that T(1) = n0 = 2 for N = 1

• So T(N) = 2N+2 is O(N) for last

46
example.