Data Structures
Fall 2024
2. Complexity Analysis
Comparing Algorithms
• Given two or more algorithms to solve the same problem,
how do we select the best one?
• Some criteria for selecting an algorithm
– Is it easy to implement, understand, modify?
– How long does it take to run it to completion?
– How much of computer memory does it use?
• Software engineering is primarily concerned with the first
criteria
• In this course we are interested in the second and third
criteria
Data Structures 1.2 - Complexity Analysis 2
Comparing Algorithms
• Time complexity
– The amount of time that an algorithm needs to run to
completion
– Better algorithm is the one which runs faster
⮚ Has smaller time complexity
• Space complexity
– The amount of memory an algorithm needs to run
• In this lecture, we will focus on analysis of time complexity
Data Structures 1.2 - Complexity Analysis 3
How To Calculate Running Time
• Most algorithms transform input objects into output
objects
sorting
5 3 1 2 1 2 3 5
algorith
m
input output
object object
sorting
5 3 1 2 4 6 1 2 3 4 5 6
algorit
hm
• The running time of an algorithm typically grows with input
size
– Idea: analyze running time as a function of input size
Data Structures 1.2 - Complexity Analysis 4
How To Calculate Running Time
• Most important factor affecting running time is usually the
size of the input
int find_max( int *array, int n ) {
int max = array[0];
for ( int i = 1; i < n; ++i ) {
if ( array[i] > max ) {
max = array[i];
}
}
return max;
}
• Regardless of the size n of an array the time complexity
will always be same
– Every element in the array is checked one time
Data Structures 1.2 - Complexity Analysis 5
How To Calculate Running Time
• Even on inputs of the same size, running time can be very
different
int search(int arr[], int n, int x) {
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
• Example: Search for 1
– Best case: Loop runs 1 times
1 2 3 4 5 6
Data Structures 1.2 - Complexity Analysis 6
How To Calculate Running Time
• Even on inputs of the same size, running time can be very
different
int search(int arr[], int n, int x) {
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
• Example: Search for 1
– Worst case: Loop runs n times
6 5 4 3 2 1
Data Structures 1.2 - Complexity Analysis 7
How To Calculate Running Time
• Even on inputs of the same size, running time can be very
different
int search(int arr[], int n, int x) {
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
• Example: Search for 1
– Average case: Loop runs between 1 and n times
3 2 1 4 5 6
Data Structures 1.2 - Complexity Analysis 8
How To Calculate Running Time
• Even on inputs of the same size, running time can be very
different
int search(int arr[], int n, int x) {
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
• Idea: Analyze running time for different cases
– Best case
– Worst case
– Average case
Data Structures 1.2 - Complexity Analysis 9
How To Calculate Running Time
• Best case running time is
usually not very useful
• Average case time is very
useful but often hard to
determine
• Worst case running time is
easier to analyze
– Crucial for real-time
applications such as games,
finance and robotics
Data Structures 1.2 - Complexity Analysis 10
Experimental Evaluations of Running Times
• Write a program
implementing the algorithm
• Run the program with inputs
of varying size
• Use clock methods to get an
accurate measure of the
actual running time
• Plot the results
Data Structures 1.2 - Complexity Analysis 11
Limitations Of Experiments
Experimental evaluation of running time is very useful but
• It is necessary to implement the algorithm, which may be
difficult
• Results may not be indicative of the running time on other
inputs not included in the experiment
• In order to compare two algorithms, the same hardware
and software environments must be used
Data Structures 1.2 - Complexity Analysis 12
Theoretical Analysis of Running Time
• Uses a pseudo-code description of the algorithm instead of
an implementation
• Characterizes running time as a function of the input size n
• Takes into account all possible inputs
• Allows us to evaluate the speed of an algorithm
independent of the hardware/software environment
Data Structures 1.2 - Complexity Analysis 13
Analyzing an Algorithm – Operations
• Each machine instruction is executed in a fixed number of
cycles
– We may assume each operation requires a fixed number of
cycles
• Idea: Use abstract machine that uses steps of time instead
of secs
– Each elementary operation takes 1 steps
• Example of operations
– Retrieving/storing variables from memory
– Variable assignment =
– Integer operations + - * / % ++ --
– Logical operations && || !
– Bitwise operations &|^~
– Relational operations == != < <= => >
Data Structures
– Memory allocation 1.2 - Complexity Analysis
and deallocation new delete 14
Analyzing an Algorithm
// Input: int A[N], array of N
integers
// Output: Sum of all numbers in
array A
1
int SumArray(int A[], int n){
int s=0;
2 3 4
for (int i=0; i< n; i++)
5
6
s = s + A[i]; 7
8
return s;
• Operations} 1, 2, and 8 are executed once
• Operations 4, 5, 6, and 7: Once per each iteration of for loop n
iteration
• Operation 3 is executed n+1 times
• The complexity function of the algorithm is : T(n) = 5n +4
Data Structures 1.2 - Complexity Analysis 15
Analyzing an Algorithm – Growth Rate
• Estimated running time for different values of n:
– n = 10 => 54 steps
– n = 100 => 504 steps
– n = 1,000 => 5004 steps
– n = 1,000,000 => 5,000,004 steps
• As n grows, number of steps T(n) grow in linear proportion
to n
Data Structures 1.2 - Complexity Analysis 16
Growth Rate
Data Structures 1.2 - Complexity Analysis 17
Growth Rate
Data Structures 1.2 - Complexity Analysis 18
Growth Rate
• Changing the hardware/software environment
– Affects T(n) by a constant factor, but
– Does not alter the growth rate of T(n)
• Thus we focus on the big-picture which is the growth rate
of an algorithm
• The linear growth rate of the running time T(n) is an
intrinsic property of algorithm sumArray
Data Structures 1.2 - Complexity Analysis 19
Constant Factors
• The growth rate is not affected by
– Constant factors or
– Lower-order terms
• Example:
– f(n) = n2
– g(n) = n2 – 3n + 2
Data Structures 1.2 - Complexity Analysis 20
Growth Rate – Example
• Consider the two functions
– f(n) = n2
– g(n) = n2 – 3n + 2
• Around n = 0, they look very different
Data Structures 1.2 - Complexity Analysis 21
Growth Rate – Example
• Yet on the range n = [0, 1000], f(n) and g(n) are
(relatively) indistinguishable
Data Structures 1.2 - Complexity Analysis 22
Growth Rate – Example
• The absolute difference is large, for example,
– f(1000) = 1 000 000
– g(1000) = 997 002
• But the relative difference is very small
– The difference goes to zero as n → ∞
Data Structures 1.2 - Complexity Analysis 23
Constant Factors
• The growth rate is not affected by
– Constant factors or
– Lower-order terms
• Example:
– f(n) = n2
– g(n) = n2 – 3n + 2
• How do we get rid of the constant factors to focus on the
essential part of the running time?
– Asymptotic Analysis
Data Structures 1.2 - Complexity Analysis 24
Upper Bound – Big-O Notation
• Indicates the upper or highest growth rate that the
algorithm can have
– Ignore constant factors and lower order terms
– Focus on main components of a function which affect its
growth
• Examples
– 55 = O(1)
– 25c + 32k = O(1) // if c,k are constants
– 5n + 6 = O(n)
– n2 – 3n + 2 = O(n2)
– 7n + 2nlog(5n) = O(nlogn)
Data Structures 1.2 - Complexity Analysis 25
Analyzing an Algorithm
• Simple Assignment
– a = b
– O(1) // Constant time complexity
• Simple loops
– for (i=0; i<n; i++) { s; }
– O(n) // Linear time complexity
• Nested loops
– for (i=0; i<n; i++)
for (j=0; j<n; j++) { s; }
– O(n2) // Quadratic time complexity
Data Structures 1.2 - Complexity Analysis 26
Analyzing an Algorithm
• Loop index doesn’t vary linearly
– h = 1;
while ( h <= n ) {
s;
h = 2 * h;
}
– h takes values 1, 2, 4, … until it exceeds n
– There are 1 + log2n iterations
– O(log n) // Logarithmic time
complexity
Data Structures 1.2 - Complexity Analysis 27
Analyzing an Algorithm
•
Data Structures 1.2 - Complexity Analysis 28
Exercises in Time Complexity Analysis
Data Structures 1.2 - Complexity Analysis 29
Final Notes
•
Running
time
A
1000
0
problem
size
Data Structures 1.2 - Complexity Analysis 30
Any Question So Far?
Data Structures 1.2 - Complexity Analysis 31
