Course Code DESIGN AND ANALYSIS OF ALGORITHMS L T P C

CS 3302 3 0 2 4

COURSE OBJECTIVES:
1. To understand and apply the algorithm analysis techniques on searching and
sorting algorithms
2. To understand the different Greedy Algorithms
3. To understand different algorithm design techniques.
4. To solve programming problems using state space tree.
5. To understand the concepts behind NP Completeness, Approximation
algorithms and randomize algorithms.

UNIT I INTRODUCTION 9
Problem Solving: Programs and Algorithms – Problem Solving Aspects – Problem
Solving Techniques - Algorithm analysis: Time and space complexity - Asymptotic
Notations and its properties Best case, Worst case and average case analysis –
Recurrence relation: substitution method and searching: Interpolation Search,
Pattern search: The naïve string- matching algorithm - Rabin-Karp algorithm -
Knuth-Morris-Pratt algorithm.

*****************************************************************
Algorithm : Systematic logical approach which is a well-defined, step-by-step
procedure that allows a computer to solve a problem.
Pseudocode : It is a simpler version of a programming code in plain English which
uses short phrases to write code for a program before it is implemented in a specific
programming language.
Program : It is exact code written for problem following all the rules of the
programming language.
Algorithm:
An algorithm is used to provide a solution to a particular problem in form of well-
defined steps. Whenever you use a computer to solve a particular problem, the steps
which lead to the solution should be properly communicated to the computer. While
executing an algorithm on a computer, several operations such as additions and
subtractions are combined to perform more complex mathematical operations.
Algorithms can be expressed using natural language, flowcharts, etc.
Algorithm of linear search:
1. Start from the leftmost element of arr[] and one by one compare x with each
element of arr[].
2. If x matches with an element, return the index.
3. If x doesn’t match with any of elements, return -1.
Pseudocode:
It is one of the methods which can be used to represent an algorithm for a program. It
does not have a specific syntax like any of the programming languages and thus
cannot be executed on a computer. There are several formats which are used to write
pseudo-codes and most of them take down the structures from languages such as C,
Lisp, FORTRAN, etc.
Many time algorithms are presented using pseudocode since they can be read and
understood by programmers who are familiar with different programming languages.
Pseudocode allows you to include several control structures such as While, If-then-
else, Repeat-until, for and case, which is present in many high-level languages.
Note: Pseudocode is not an actual programming language.
Pseudocode for Linear Search:
FUNCTION linearSearch(list, searchTerm):
FOR index FROM 0 -> length(list):
IF list[index] == searchTerm THEN
RETURN index
ENDIF
ENDLOOP
RETURN -1
END FUNCTION
 Program:
A program is a set of instructions for the computer to follow. The machine can’t read
a program directly, because it only understands machine code. But you can write stuff
in a computer language, and then a compiler or interpreter can make it understandable
to the computer.
Program for Linear Search :
// C++ code for linearly search x in arr[]. If x
// is present then return its location, otherwise
// return -1
int search(int arr[], int n, int x)
{
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
Algorithm vs Pseudocode vs Program:
An algorithm is defined as a well-defined sequence of steps that provides a solution
for a given problem, whereas a pseudocode is one of the methods that can be used to
represent an algorithm.
While algorithms are generally written in a natural language or plain English
language, pseudocode is written in a format that is similar to the structure of a high-
level programming language. Program on the other hand allows us to write a code in
a particular programming language.
PROBLEM SOLVING:
Problem solving is the act of defining a problem; determining the cause of the problem; identifying,
prioritizing, and selecting alternatives for a solution; and implementing a solution.
The problem-solving process
Problem solving resources

THE PROBLEM-SOLVING PROCESS

In order to effectively manage and run a successful organization, leadership must

guide their employees and develop problem-solving techniques. Finding a suitable
solution for issues can be accomplished by following the basic four-step problem-
solving process and methodology outlined below.

Step Characteristics

1. Define the  Differentiate fact from opinion

problem  Specify underlying causes
 Consult each faction involved for information
 State the problem specifically
 Identify what standard or expectation is violated
 Determine in which process the problem lies
 Avoid trying to solve the problem without data

2. Generate  Postpone evaluating alternatives initially

alternative solutions  Include all involved individuals in the generating of alternatives
 Specify alternatives consistent with organizational goals
 Specify short- and long-term alternatives
 Brainstorm on others' ideas
 Seek alternatives that may solve the problem

3. Evaluate and  Evaluate alternatives relative to a target standard

select an alternative  Evaluate all alternatives without bias
 Evaluate alternatives relative to established goals
 Evaluate both proven and possible outcomes
 Step Characteristics

 State the selected alternative explicitly

4. Implement and  Plan and implement a pilot test of the chosen alternative
follow up on the  Gather feedback from all affected parties
solution  Seek acceptance or consensus by all those affected
 Establish ongoing measures and monitoring
 Evaluate long-term results based on final solution

1. Define the problem

Diagnose the situation so that your focus is on the problem, not just its symptoms.
Helpful problem-solving techniques include using flowcharts to identify the expected
steps of a process and cause-and-effect diagrams to define and analyze root causes.

The sections below help explain key problem-solving steps. These steps support the
involvement of interested parties, the use of factual information, comparison of
expectations to reality, and a focus on root causes of a problem. You should begin
by:

 Reviewing and documenting how processes currently work (i.e., who does
what, with what information, using what tools, communicating with what
organizations and individuals, in what time frame, using what format).
 Evaluating the possible impact of new tools and revised policies in the
development of your "what should be" model.

2. Generate alternative solutions

Postpone the selection of one solution until several problem-solving alternatives have
been proposed. Considering multiple alternatives can significantly enhance the value
of your ideal solution. Once you have decided on the "what should be" model, this
target standard becomes the basis for developing a road map for investigating
alternatives. Brainstorming and team problem-solving techniques are both useful
tools in this stage of problem solving.

Many alternative solutions to the problem should be generated before final

evaluation. A common mistake in problem solving is that alternatives are evaluated
as they are proposed, so the first acceptable solution is chosen, even if it’s not the
best fit. If we focus on trying to get the results we want, we miss the potential for
learning something new that will allow for real improvement in the problem-solving
process.
3. Evaluate and select an alternative

Skilled problem solvers use a series of considerations when selecting the best
alternative. They consider the extent to which:

 A particular alternative will solve the problem without causing other

unanticipated problems.
 All the individuals involved will accept the alternative.
 Implementation of the alternative is likely.
 The alternative fits within the organizational constraints.

4. Implement and follow up on the solution

Leaders may be called upon to direct others to implement the solution, "sell" the
solution, or facilitate the implementation with the help of others. Involving others in
the implementation is an effective way to gain buy-in and support and minimize
resistance to subsequent changes.

Regardless of how the solution is rolled out, feedback channels should be built into
the implementation. This allows for continuous monitoring and testing of actual
events against expectations. Problem solving, and the techniques used to gain clarity,
are most effective if the solution remains in place and is updated to respond to future
changes.

ALGORITHMIC PROBLEM SOLVING:

Algorithmic problem solving is solving problem that require the formulation of an
algorithm for the solution.
Understanding the Problem
 It is the process of finding the input of the problem that the algorithm solves
 It is very important to specify exactly the set of inputs the algorithm needs to
handle.
 A correct algorithm is not one that works most of the time, but one that works
correctly for all legitimate inputs.
Ascertaining the Capabilities of the Computational Device
 If the instructions are executed one after another, it is called sequential
algorithm.
 If the instructions are executed concurrently, it is called parallel algorithm.
Choosing between Exact and Approximate Problem Solving
 The next principal decision is to choose between solving the problem exactly or
solving it approximately.
 Based on this, the algorithms are classified as
exact algorithm and approximation algorithm.
Deciding a data structure:
 Data structure plays a vital role in designing and analysis the algorithms.
 Some of the algorithm design techniques also depend on the structuring data
specifying a problem’s instance
 Algorithm+ Data structure=programs.
Algorithm Design Techniques
 An algorithm design technique (or “strategy” or “paradigm”) is a general
approach to solving problems algorithmically that is applicable to a variety of
problems from different areas of computing.
 Learning these techniques is of utmost importance for the following reasons.
 First, they provide guidance for designing algorithms for new problems,
 Second, algorithms are the cornerstone of computer science
Methods of Specifying an Algorithm
 Pseudocode is a mixture of a natural language and programming language-
like constructs. Pseudocode is usually more precise than natural language, and
its usage often yields more succinct algorithm descriptions.
 In the earlier days of computing, the dominant vehicle for specifying algorithms
was a flowchart, a method of expressing an algorithm by a collection of
connected geometric shapes containing descriptions of the algorithm’s steps.
 Programming language can be fed into an electronic computer directly.
Instead, it needs to be converted into a computer program written in a particular
computer language. We can look at such a program as yet another way
of specifying the algorithm, although it is preferable to consider it as the
algorithm’s implementation.
Proving an Algorithm’s Correctness
 Once an algorithm has been specified, you have to prove its correctness. That
 is, you have to prove that the algorithm yields a required result for every
legitimate input in a finite amount of time.
 A common technique for proving correctness is to use mathematical induction
because an algorithm’s iterations provide a natural sequence of steps needed for
such proofs.
 It might be worth mentioning that although tracing the algorithm’s performance
for a few specific inputs can be a very worthwhile activity, it cannot prove the
algorithm’s correctness conclusively. But in order to show that an algorithm is
incorrect, you need just one instance of its input for which the algorithm fails.
Analysing an Algorithm
1. Efficiency.
Time efficiency, indicating how fast the algorithm runs,
Space efficiency, indicating how much extra memory it uses.

2. simplicity.
 An algorithm should be precisely defined and investigated with mathematical
expressions.
 Simpler algorithms are easier to understand and easier to program.
 Simple algorithms usually contain fewer bugs.
Coding an Algorithm
 Most algorithms are destined to be ultimately implemented as computer
programs. Programming an algorithm presents both a peril and an opportunity.
 A working program provides an additional opportunity in allowing an empirical
analysis of the underlying algorithm. Such an analysis is based on timing the
program on several inputs and then analysing the results obtained.

Algorithm Analysis
Time Complexity: The time complexity of an algorithm quantifies the amount of
time taken by an algorithm to run as a function of the length of the input. Note that
the time to run is a function of the length of the input and not the actual execution
time of the machine on which the algorithm is running on.
Definition:
The valid algorithm takes a finite amount of time for execution. The time required
by the algorithm to solve given problem is called time complexity of the algorithm.
Time complexity is very useful measure in algorithm analysis.
Example 1: Addition of two scalar variables.
Algorithm ADD SCALAR(A, B)
//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <- A + B
return C
The addition of two scalar numbers requires one addition operation. the time
complexity of this algorithm is constant, so T(n) = O(1) .

In order to calculate time complexity on an algorithm, it is assumed that a constant

time c is taken to execute one operation, and then the total operations for an input
length on N are calculated.
Space Complexity:
Definition – Problem-solving using computer requires memory to hold temporary
data or final result while the program is in execution. The amount of memory required
by the algorithm to solve given problem is called space complexity of the algorithm.

The space complexity of an algorithm quantifies the amount of space taken by an

algorithm to run as a function of the length of the input. Consider an example:
Suppose a problem to find the frequency of array elements.

It is the amount of memory needed for the completion of an algorithm.

To estimate the memory requirement we need to focus on two parts:

(1) A fixed part: It is independent of the input size. It includes memory for instructions
(code), constants, variables, etc.

(2) A variable part: It is dependent on the input size. It includes memory for recursion
stack, referenced variables, etc.

Example : Addition of two scalar variables

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <— A+B
return C
The addition of two scalar numbers requires one extra memory location to hold the
result. Thus the space complexity of this algorithm is constant, hence S(n) = O(1).

Complexity analysis is defined as a technique to characterize the time taken

by an algorithm with respect to input size (independent from the machine,
language and compiler). It is used for evaluating the variations of execution time on
different algorithms.

Need of Complexity Analysis:

● Complexity Analysis determines the amount of time and space resources
required to execute it.
● It is used for comparing different algorithms on different input sizes.
 ● Complexity helps to determine the difficulty of a problem.
● often measured by how much time and space (memory) it takes to solve a
particular problem

Asymptotic Notations in Complexity Analysis:

1. Big O Notation
Big-O notation represents the upper bound of the running time of an
algorithm. Therefore, it gives the worst-case complexity of an algorithm. By using
big O- notation, we can asymptotically limit the expansion of a running time to a
range of constant factors above and below. It is a model for quantifying algorithm
performance.

Mathematical Representation of Big-O Notation:

O(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n)
for all n ≥ n0 }

2. Omega Notation
Omega notation represents the lower bound of the running time of an
algorithm. Thus, it provides the best-case complexity of an algorithm.
The execution time serves as a lower bound on the algorithm’s time complexity. It
is defined as the condition that allows an algorithm to complete statement execution
in the shortest amount of time.
Mathematical Representation of Omega Notation:
Ω(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n)
for all n ≥ n0 }
Note: Ω (g) is a set

3. Theta Notation
Theta notation encloses the function from above and below. Since it represents
the upper and the lower bound of the running time of an algorithm, it is used for
analyzing the average-case complexity of an algorithm. The execution time serves
as both a lower and upper bound on the algorithm’s time complexity. It exists as both,
the most, and least boundaries for a given input value.

Mathematical Representation:
Θ (g(n)) = {f(n): there exist positive constants c1, c2 and n0 such that 0 ≤ c1 * g(n)
≤ f(n) ≤ c2 * g(n) for all n ≥ n0}

4. Little ο asymptotic notation

Big-Ο is used as a tight upper bound on the growth of an algorithm’s effort
(this effort is described by the function f(n)), even though, as written, it can also be
a loose upper bound. “Little-ο” (ο()) notation is used to describe an upper bound that
cannot be tight.

Mathematical Representation:
f(n) = o(g(n)) means lim f(n)/g(n) = 0 n→∞
5. Little ω asymptotic notation
Let f(n) and g(n) be functions that map positive integers to positive real
numbers. We say that f(n) is ω(g(n)) (or f(n) ∈ ω(g(n))) if for any real constant c >
0, there exists an integer constant n0 ≥ 1 such that f(n) > c * g(n) ≥ 0 for every integer
n ≥ n0.
Mathematical Representation:
if f(n) ∈ ω(g(n)) then,
lim f(n)/g(n) = ∞
n→∞

Measure of Complexity:
The complexity of an algorithm can be measured in three ways:

1. Time Complexity
The time complexity of an algorithm is defined as the amount of time taken
by an algorithm to run as a function of the length of the input. Note that the time to
run is a function of the length of the input and not the actual execution time of the
machine on which the algorithm is running on

How is Time complexity computed?

To estimate the time complexity, we need to consider the cost of each

fundamental instruction and the number of times the instruction is executed.

If we have statements with basic operations like comparisons, return

statements, assignments, and reading a variable. We can assume they take constant
time each O(1).
Statement 1: int a=5; // reading a variable
statement 2; if( a==5) return true; // return statement
statement 3; int x= 4>5 ? 1:0; // comparison
statement 4; bool flag=true; // Assignment

This is the result of calculating the overall time complexity.

total time = time(statement1) + time(statement2) + ... time (statementN)

Assuming that n is the size of the input, let’s use T(n) to represent the overall
time and t to represent the amount of time that a statement or collection of statements
takes to execute.

T(n) = t(statement1) + t(statement2) + ... + t(statementN);

Overall, T(n)= O(1), which means constant complexity.

 For any loop, we find out the runtime of the block inside them and multiply
it by the number of times the program will repeat the loop.
for (int i = 0; i < n; i++) {
cout << “GeeksForGeeks” << endl;
}

For the above example, the loop will execute n times, and it will print
“GeeksForGeeks” N number of times. so the time taken to run this program is:

T(N)= n *( t(cout statement))

= n * O(1)
=O(n), Linear complexity.

For 2D arrays, we would have nested loop concepts, which means a loop
inside a loop.
for (int i = 0; i < n; i++) {
for (int j = 0; j < m; j++) {
cout << “GeeksForGeeks” << endl;
}
}

For the above example, the cout statement will execute n*m times, and it will
print “GeeksForGeeks” N*M number of times. so the time taken to run this program
is:

T(N)= n * m *(t(cout statement))

= n * m * O(1)
=O(n*m), Quadratic Complexity.

2. Space Complexity :
The amount of memory required by the algorithm to solve a given problem is
called the space complexity of the algorithm. Problem-solving using a computer
requires memory to hold temporary data or final result while the program is in
execution.

How is space complexity computed?

The space Complexity of an algorithm is the total space taken by the algorithm
with respect to the input size. Space complexity includes both Auxiliary space and
space used by input.
Space complexity is a parallel concept to time complexity. If we need to create
an array of size n, this will require O(n) space. If we create a two-
dimensional array of size n*n, this will require O(n2) space.

In recursive calls stack space also counts.

Example:

int add (int n){

if (n <= 0){
return 0;
}
return n + add (n-1);
}
Here each call add a level to the stack :
1. add(4)
2. -> add(3)
3. -> add(2)
4. -> add(1)
5. -> add(0)
Each of these calls is added to call stack and takes up actual memory.
So it takes O(n) space.
However, just because you have n calls total doesn’t mean it takes O(n)
space.

Look at the below function :

int addSequence (int n){

int sum = 0;
for (int i = 0; i < n; i++){
sum += pairSum(i, i+1);
}
return sum;
}
int pairSum(int x, int y){
return x + y;
}
There will be roughly O(n) calls to pairSum. However, those calls do not
exist simultaneously on the call stack, so you only need O(1) space.

3. Auxiliary Space :

The temporary space needed for the use of an algorithm is referred to as

auxiliary space. Like temporary arrays, pointers, etc.
It is preferable to make use of Auxiliary Space when comparing things like sorting
algorithms. For example, sorting algorithms take O(n) space, as there is an input
array to sort. but the auxiliary space is O(1) in that case.

Optimizing the time and space complexity of an algorithm:

Optimization means modifying the brute-force approach to a problem. It is
done to derive the best possible solution to solve the problem so that it will take less
time and space complexity. We can optimize a program by either limiting the search
space at each step or occupying less search space from the start.
We can optimize a solution using both time and space optimization. To optimize a
program,
● We can reduce the time taken to run the program and increase the space
occupied;
● we can reduce the memory usage of the program and increase its total run
time, or
● we can reduce both time and space complexity by deploying relevant
algorithms

Time Complexity:
The valid algorithm takes a finite amount of time for execution. The time
required by the algorithm to solve given problem is called time complexity of the
algorithm. Time complexity is very useful measure in algorithm analysis. It is the
time needed for the completion of an algorithm. To estimate the time complexity,
we need to consider the cost of each fundamental instruction and the number of times
the instruction is executed.

Example 1: Addition of two scalar variables.

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <- A + B
return C
The addition of two scalar numbers requires one addition operation. the time
complexity of this algorithm is constant, so T(n) = O(1) .

In order to calculate time complexity on an algorithm, it is assumed that a

constant time c is taken to execute one operation, and then the total operations for
an input length on N are calculated. Consider an example to understand the process
of calculation: Suppose a problem is to find whether a pair (X, Y) exists in an array,
A of N elements whose sum is Z. The simplest idea is to consider every pair and
check if it satisfies the given condition or not.

The pseudo-code is as follows:

int a[n];
for(int i = 0;i < n;i++)
cin >> a[i]

for(int i = 0;i < n;i++)

for(int j = 0;j < n;j++)
if(i!=j && a[i]+a[j] == z)
return true
return false
Assuming that each of the operations in the computer takes approximately
constant time, let it be c. The number of lines of code executed actually depends on
the value of Z. During analyses of the algorithm, mostly the worst-case scenario is
considered, i.e., when there is no pair of elements with sum equals Z. In the worst
case,

N*c operations are required for input.

The outer loop i loop runs N times.
For each i, the inner loop j loop runs N times.
So total execution time is N*c + N*N*c + c. Now ignore the lower order terms
since the lower order terms are relatively insignificant for large input, therefore only
the highest order term is taken (without constant) which is N*N in this case.
Different notations are used to describe the limiting behavior of a
function, but since the worst case is taken so big-O notation will be used to
represent the time complexity.

Hence, the time complexity is O(N2) for the above algorithm. Note that the
time complexity is solely based on the number of elements in array A i.e the input
length, so if the length of the array will increase the time of execution will also
increase.

Order of growth is how the time of execution depends on the length of the
input. In the above example, it is clearly evident that the time of execution
quadratically depends on the length of the array. Order of growth will help to
compute the running time with ease.
Some general time complexities are listed below with the input range for
which they are accepted in competitive programming:

Worst Accepted Usually type of solutions

Input Length
Time Complexity

10 -12 O(N!) Recursion and backtracking

Recursion, backtracking, and bit

15-18 O(2N * N)
manipulation

Recursion, backtracking, and bit

18-22 O(2N * N)
manipulation
30-40 Meet in the middle, Divide and Conquer
O(2N/2 * N)

100 O(N4) Dynamic programming, Constructive

400 O(N3) Dynamic programming, Constructive

Dynamic programming, Binary Search,

2K O(N2* log N) Sorting,
Divide and Conquer

Dynamic programming, Graph, Trees,

10K O(N2)
Constructive

Sorting, Binary Search, Divide and

1M O(N* log N)
Conquer
 O(N), O(log N), Constructive, Mathematical, Greedy
100M
O(1) Algorithms

Space Complexity:
Problem-solving using computer requires memory to hold temporary data or
final result while the program is in execution. The amount of memory required by
the algorithm to solve given problem is called space complexity of the algorithm.

The space complexity of an algorithm quantifies the amount of space taken

by an algorithm to run as a function of the length of the input. Consider an example:
Suppose a problem to find the frequency of array elements.

It is the amount of memory needed for the completion of an algorithm.

To estimate the memory requirement we need to focus on two parts:

(1) A fixed part: It is independent of the input size. It includes memory for
instructions (code), constants, variables, etc.

(2) A variable part: It is dependent on the input size. It includes memory for
recursion stack, referenced variables, etc.

Example : Addition of two scalar variables

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <— A+B
return C
The addition of two scalar numbers requires one extra memory location to
hold the result. Thus the space complexity of this algorithm is constant, hence S(n)
= O(1).

The pseudo-code is as follows:

int freq[n];
int a[n];
for(int i = 0; i<n; i++)
{
cin>>a[i];
freq[a[i]]++;
}
There is also auxiliary space, which is different from space complexity. The
main difference is where space complexity quantifies the total space used by the
algorithm, auxiliary space quantifies the extra space that is used in the algorithm
apart from the given input. In the above example, the auxiliary space is the space
used by the freq[] array because that is not part of the given input. So total auxiliary
space is N * c + c which is O(N) only.

Asymptotic Notations:

Asymptotic Notations are mathematical tools used to analyze the performance

of algorithms by understanding how their efficiency changes as the input size grows.
These notations provide a concise way to express the behavior of an algorithm’s time
or space complexity as the input size approaches infinity. Rather than comparing
algorithms directly, asymptotic analysis focuses on understanding the relative
growth rates of algorithms’ complexities. It enables comparisons of algorithms’
efficiency by abstracting away machine-specific constants and implementation
details, focusing instead on fundamental trends.
Asymptotic analysis allows for the comparison of algorithms’ space and time
complexities by examining their performance characteristics as the input size varies.
By using asymptotic notations, such as Big O, Big Omega, and Big Theta, we can
categorize algorithms based on their worst-case, best-case, or average-case time or
space complexities, providing valuable insights into their efficiency.

There are mainly three asymptotic notations:

1. Big-O Notation (O-notation)
2. Omega Notation (Ω-notation)
3. Theta Notation (Θ-notation)

1. Theta Notation (Θ-Notation):

Theta notation encloses the function from above and below. Since it
represents the upper and the lower bound of the running time of an algorithm, it is
used for analyzing the average-case complexity of an algorithm.
 Theta (Average Case) You add the running times for each possible input
combination and take the average in the average case.

Let g and f be the function from the set of natural numbers to itself. The
function f is said to be Θ(g), if there are constants c1, c2 > 0 and a natural number
n0 such that c1* g(n) ≤ f(n) ≤ c2 * g(n) for all n ≥ n0

Mathematical Representation of Theta notation:

Θ (g(n)) = {f(n): there exist positive constants c1, c2 and n0 such that 0 ≤ c1 * g(n)
≤ f(n) ≤ c2 * g(n) for all n ≥ n0}

Note: Θ(g) is a set

The above expression can be described as if f(n) is theta of g(n), then the value
f(n) is always between c1 * g(n) and c2 * g(n) for large values of n (n ≥ n0). The
definition of theta also requires that f(n) must be non-negative for values of n greater
than n0.

The execution time serves as both a lower and upper bound on the algorithm’s
time complexity.

It exist as both, most, and least boundaries for a given input value.

A simple way to get the Theta notation of an expression is to drop low-order

terms and ignore leading constants. For example, Consider the expression 3n3 + 6n2
+ 6000 = Θ(n3), the dropping lower order terms is always fine because there will
always be a number(n) after which Θ(n3) has higher values than Θ(n2) irrespective
of the constants involved. For a given function g(n), we denote Θ(g(n)) is following
set of functions.

Examples :

{ 100 , log (2000) , 10^4 } belongs to Θ(1)

{ (n/4) , (2n+3) , (n/100 + log(n)) } belongs to Θ(n)
{ (n^2+n) , (2n^2) , (n^2+log(n))} belongs to Θ( n2)

Note: Θ provides exact bounds.

2. Big-O Notation (O-notation):

 Big-O notation represents the upper bound of the running time of an
algorithm. Therefore, it gives the worst-case complexity of an algorithm.

● It is the most widely used notation for Asymptotic analysis.

● It specifies the upper bound of a function.
● The maximum time required by an algorithm or the worst-case time
complexity.
● It returns the highest possible output value(big-O) for a given input.
● Big-Oh(Worst Case) It is defined as the condition that allows an algorithm to
complete statement execution in the longest amount of time possible.
If f(n) describes the running time of an algorithm, f(n) is O(g(n)) if there
exist a positive constant C and n0 such that, 0 ≤ f(n) ≤ cg(n) for all n ≥ n0

It returns the highest possible output value (big-O)for a given input.

The execution time serves as an upper bound on the algorithm’s time

complexity.

Mathematical Representation of Big-O Notation:

O(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for
all n ≥ n0 }

For example, Consider the case of Insertion Sort. It takes linear time in the
best case and quadratic time in the worst case. We can safely say that the time
complexity of the Insertion sort is O(n2).
Note: O(n2) also covers linear time.

If we use Θ notation to represent the time complexity of Insertion sort, we

have to use two statements for best and worst cases:

● The worst-case time complexity of Insertion Sort is Θ(n2).

● The best case time complexity of Insertion Sort is Θ(n).
● The Big-O notation is useful when we only have an upper bound on the time
complexity of an algorithm. Many times we easily find an upper bound by
simply looking at the algorithm.

Examples :

{ 100 , log (2000) , 10^4 } belongs to O(1)

U { (n/4) , (2n+3) , (n/100 + log(n)) } belongs to O(n)
U { (n^2+n) , (2n^2) , (n^2+log(n))} belongs to O( n^2)

Note: Here, U represents union, we can write it in these manner because O

provides exact or upper bounds .

3. Omega Notation (Ω-Notation):

Omega notation represents the lower bound of the running time of an
algorithm. Thus, it provides the best case complexity of an algorithm.

The execution time serves as a lower bound on the algorithm’s time

complexity.

It is defined as the condition that allows an algorithm to complete statement

execution in the shortest amount of time.

Let g and f be the function from the set of natural numbers to itself. The
function f is said to be Ω(g), if there is a constant c > 0 and a natural number n0 such
that c*g(n) ≤ f(n) for all n ≥ n0

Mathematical Representation of Omega notation :

Ω(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n)
for all n ≥ n0 }

Let us consider the same Insertion sort example here. The time complexity of
Insertion Sort can be written as Ω(n), but it is not very useful information about
insertion sort, as we are generally interested in worst-case and sometimes in the
average case.

Examples :

{ (n^2+n) , (2n^2) , (n^2+log(n))} belongs to Ω( n^2)

U { (n/4) , (2n+3) , (n/100 + log(n)) } belongs to Ω(n)
U { 100 , log (2000) , 10^4 } belongs to Ω(1)

Note: Here, U represents union, we can write it in these manner because Ω

provides exact or lower bounds.

Properties of Asymptotic Notations:

1. General Properties:
If f(n) is O(g(n)) then a*f(n) is also O(g(n)), where a is a constant.

Example:

f(n) = 2n²+5 is O(n²)

then, 7*f(n) = 7(2n²+5) = 14n²+35 is also O(n²).

Similarly, this property satisfies both Θ and Ω notation.

We can say,

If f(n) is Θ(g(n)) then a*f(n) is also Θ(g(n)), where a is a constant.

If f(n) is Ω (g(n)) then a*f(n) is also Ω (g(n)), where a is a constant.

2. Transitive Properties:
If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) = O(h(n)).
f(n) = O(g(n)) and g(n) = O(h(n)) ⇒ f(n) = O(h(n))

Example:

If f(n) = n, g(n) = n² and h(n)=n³

n is O(n²) and n² is O(n³) then, n is O(n³)

Similarly, this property satisfies both Θ and Ω notation.

We can say,

If f(n) is Θ(g(n)) and g(n) is Θ(h(n)) then f(n) = Θ(h(n)) .

If f(n) is Ω (g(n)) and g(n) is Ω (h(n)) then f(n) = Ω (h(n))

3. Reflexive Properties:
Reflexive properties are always easy to understand after transitive.
If f(n) is given then f(n) is O(f(n)). Since MAXIMUM VALUE OF f(n) will be f(n)
ITSELF!
f(n) = O(f(n))
Hence x = f(n) and y = O(f(n) tie themselves in reflexive relation always.

Example:
f(n) = n² ; O(n²) i.e O(f(n))
Similarly, this property satisfies both Θ and Ω notation.

We can say that,

If f(n) is given then f(n) is Θ(f(n)).

If f(n) is given then f(n) is Ω (f(n)).

4. Symmetric Properties:
If f(n) is Θ(g(n)) then g(n) is Θ(f(n)).
f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n))

Example:

If(n) = n² and g(n) = n²

then, f(n) = Θ(n²) and g(n) = Θ(n²)

This property only satisfies for Θ notation.

5. Transpose Symmetric Properties:

If f(n) is O(g(n)) then g(n) is Ω (f(n)).

Example:

If(n) = n , g(n) = n²
then n is O(n²) and n² is Ω (n)

This property only satisfies O and Ω notations.

6. Some More Properties:

1. If f(n) = O(g(n)) and f(n) = Ω(g(n)) then f(n) = Θ(g(n))
2. If f(n) = O(g(n)) and d(n)=O(e(n)) then f(n) + d(n) = O( max( g(n), e(n) ))

Example:

f(n) = n i.e O(n)

d(n) = n² i.e O(n²)
then f(n) + d(n) = n + n² i.e O(n²)

3. If f(n)=O(g(n)) and d(n)=O(e(n)) then f(n) * d(n) = O( g(n) * e(n))

Example:

f(n) = n i.e O(n)

d(n) = n² i.e O(n²)
then f(n) * d(n) = n * n² = n³ i.e O(n³)

Note: If f(n) = O(g(n)) then g(n) = Ω(f(n))

Measurement of Complexity of an Algorithm

Based on the above three notations of Time Complexity there are three cases
to analyze an algorithm:

1. Worst Case Analysis (Mostly used)

In the worst-case analysis, we calculate the upper bound on the running time
of an algorithm. We must know the case that causes a maximum number of
operations to be executed. For Linear Search, the worst case happens when the
element to be searched (x) is not present in the array. When x is not present, the
search() function compares it with all the elements of arr[] one by one. Therefore,
the worst-case time complexity of the linear search would be O(n).

2. Best Case Analysis (Very Rarely used)

In the best-case analysis, we calculate the lower bound on the running time of
an algorithm. We must know the case that causes a minimum number of operations
to be executed. In the linear search problem, the best case occurs when x is present
at the first location. The number of operations in the best case is constant (not
dependent on n). So time complexity in the best case would be ?(1)

3. Average Case Analysis (Rarely used)

In average case analysis, we take all possible inputs and calculate the
computing time for all of the inputs. Sum all the calculated values and divide the
sum by the total number of inputs. We must know (or predict) the distribution of
cases. For the linear search problem, let us assume that all cases are uniformly
distributed (including the case of x not being present in the array). So we sum all the
cases and divide the sum by (n+1). Following is the value of average-case time
complexity.

Average Case Time = \sum_{i=1}^{n}\frac{\theta (i)}{(n+1)} = \frac{\theta

(\frac{(n+1)*(n+2)}{2})}{(n+1)} = \theta (n)
Below is the ranked mention of complexity analysis notation based on
popularity:

1. Worst Case Analysis:

Most of the time, we do worst-case analyses to analyze algorithms. In the
worst analysis, we guarantee an upper bound on the running time of an algorithm
which is good information.

2. Average Case Analysis

The average case analysis is not easy to do in most practical cases and it is
rarely done. In the average case analysis, we must know (or predict) the
mathematical distribution of all possible inputs.

3. Best Case Analysis

The Best Case analysis is bogus. Guaranteeing a lower bound on an algorithm
doesn’t provide any information as in the worst case, an algorithm may take years
to run.

Examples:
1. Linear search algorithm:
// C++ implementation of the approach
#include <bits/stdc++.h>
using namespace std;

// Linearly search x in arr[].

// If x is present then return the index,
// otherwise return -1
int search(int arr[], int n, int x)
{
int i;
for (i = 0; i < n; i++) {
if (arr[i] == x)
return i;
}
return -1;
}

// Driver's Code
int main()
{
int arr[] = { 1, 10, 30, 15 };
int x = 30;
int n = sizeof(arr) / sizeof(arr[0]);

// Function call
cout << x << " is present at index "
<< search(arr, n, x);

return 0;
}
Output
30 is present at index 2

Time Complexity Analysis: (In Big-O notation)

Best Case: O(1), This will take place if the element to be searched is on the first
index of the given list. So, the number of comparisons, in this case, is 1.
Average Case: O(n), This will take place if the element to be searched is on the
middle index of the given list.
Worst Case: O(n), This will take place if:
The element to be searched is on the last index
The element to be searched is not present on the list

Advantages:
● This technique allows developers to understand the performance of algorithms
under different scenarios, which can help in making informed decisions about
which algorithm to use for a specific task.
● Worst case analysis provides a guarantee on the upper bound of the running
time of an algorithm, which can help in designing reliable and efficient
algorithms.
● Average case analysis provides a more realistic estimate of the running time
of an algorithm, which can be useful in real-world scenarios.

Disadvantages:
● This technique can be time-consuming and requires a good understanding of
the algorithm being analyzed.
● Worst case analysis does not provide any information about the typical
running time of an algorithm, which can be a disadvantage in real-world
scenarios.
 ● Average case analysis requires knowledge of the probability distribution of
input data, which may not always be available.

Important points:
● The worst case analysis of an algorithm provides an upper bound on the
running time of the algorithm for any input size.
● The average case analysis of an algorithm provides an estimate of the
running time of the algorithm for a random input.
● The best case analysis of an algorithm provides a lower bound on the
running time of the algorithm for any input size.
● The big O notation is commonly used to express the worst case running time
of an algorithm.
● Different algorithms may have different best, average, and worst case
running times.

Recurrence Relation
A recurrence is an equation or inequality that describes a function in terms of
its values on smaller inputs. To solve a Recurrence Relation means to obtain a
function defined on the natural numbers that satisfy the recurrence.

There are four methods for solving Recurrence:

1. Substitution Method
2. Iteration Method
3. Recursion Tree Method
4. Master Method

1. Substitution Method:
The Substitution Method Consists of two main steps:
● Guess the Solution.
● Use the mathematical induction to find the boundary condition and
shows that the guess is correct.

For Example Solve the equation by Substitution Method.

T (n) = TDAA Recurrence Relation + n

We have to show that it is asymptotically bound by O (log n).

Solution:

For T (n) = O (log n)

We have to show that for some constant c

T (n) ≤c logn.
Put this in given Recurrence Equation.

T (n) ≤c logDAA Recurrence Relation+ 1

≤c logDAA Recurrence Relation+ 1 = c logn-clog2 2+1
≤c logn for c≥1
Thus T (n) =O logn.
Example2 Consider the Recurrence

T (n) = 2TDAA Recurrence Relation+ n n>1

Find an Asymptotic bound on T.

Solution:

We guess the solution is O (n (logn)).Thus for constant 'c'.

T (n) ≤c n logn
Put this in given Recurrence Equation.
Now,
T (n) ≤2cDAA Recurrence Relation Recurrence Relation+n
≤cnlogn-cnlog2+n
=cn logn-n (clog2-1)
≤cn logn for (c≥1)
Thus T (n) = O (n logn).
2. Iteration Methods
It means to expand the recurrence and express it as a summation of terms of n and
initial condition.

Example1: Consider the Recurrence

T (n) = 1 if n=1
= 2T (n-1) if n>1
Solution:

T (n) = 2T (n-1)
= 2[2T (n-2)] = 22T (n-2)
= 4[2T (n-3)] = 23T (n-3)
= 8[2T (n-4)] = 24T (n-4) (Eq.1)
Repeat the procedure for i times

T (n) = 2i T (n-i)
Put n-i=1 or i= n-1 in (Eq.1)
T (n) = 2n-1 T (1)
= 2n-1 .1 {T (1) =1 ..... given}
= 2n-1
Example2: Consider the Recurrence

T (n) = T (n-1) +1 and T (1) = θ (1).

Solution:

T (n) = T (n-1) +1
= (T (n-2) +1) +1 = (T (n-3) +1) +1+1
= T (n-4) +4 = T (n-5) +1+4
= T (n-5) +5= T (n-k) + k
Where k = n-1
T (n-k) = T (1) = θ (1)
T (n) = θ (1) + (n-1) = 1+n-1=n= θ (n).

Recursion Tree Method

Recursion is a fundamental concept in computer science and mathematics that
allows functions to call themselves, enabling the solution of complex problems
through iterative steps. One visual representation commonly used to understand and
analyze the execution of recursive functions is a recursion tree. In this article, we
will explore the theory behind recursion trees, their structure, and their significance
in understanding recursive algorithms.

What is a Recursion Tree?

A recursion tree is a graphical representation that illustrates the execution flow
of a recursive function. It provides a visual breakdown of recursive calls, showcasing
the progression of the algorithm as it branches out and eventually reaches a base
case. The tree structure helps in analyzing the time complexity and understanding
the recursive process involved.

Tree Structure
Each node in a recursion tree represents a particular recursive call. The initial
call is depicted at the top, with subsequent calls branching out beneath it. The tree
grows downward, forming a hierarchical structure. The branching factor
of each node depends on the number of recursive calls made within the function.
Additionally, the depth of the tree corresponds to the number of recursive calls
before reaching the base case.

Base Case
The base case serves as the termination condition for a recursive function. It
defines the point at which the recursion stops and the function starts returning values.
In a recursion tree, the nodes representing the base case are usually depicted as leaf
nodes, as they do not result in further recursive calls.

Recursive Calls
The child nodes in a recursion tree represent the recursive calls made within
the function. Each child node corresponds to a separate recursive call, resulting in
the creation of new sub problems. The values or parameters passed to these recursive
calls may differ, leading to variations in the sub problems' characteristics.

Execution Flow:
Traversing a recursion tree provides insights into the execution flow of a
recursive function. Starting from the initial call at the root node, we follow the
branches to reach subsequent calls until we encounter the base case. As the base
cases are reached, the recursive calls start to return, and their respective nodes in the
tree are marked with the returned values. The traversal continues until the entire tree
has been traversed.

Time Complexity Analysis

Recursion trees aid in analyzing the time complexity of recursive algorithms.
By examining the structure of the tree, we can determine the number of recursive
calls made and the work done at each level. This analysis helps in understanding the
overall efficiency of the algorithm and identifying any potential inefficiencies or
opportunities for optimization.

Introduction
Think of a program that determines a number's factorial. This function takes
a number N as an input and returns the factorial of N as a result. This function's
pseudo-code will resemble,
// find factorial of a number
factorial(n) {
// Base case
if n is less than 2: // Factorial of 0, 1 is 1
return n
// Recursive step
return n * factorial(n-1); // Factorial of 5 => 5 * Factorial(4)...
}

/* How function calls are made,

Factorial(5) [ 120 ]
|
5 * Factorial(4) ==> 120
|
4. * Factorial(3) ==> 24
|
3 * Factorial(2) ==> 6
|
2 * Factorial(1) ==> 2
|
1

*/
Recursion is exemplified by the function that was previously mentioned. We
are invoking a function to determine a number's factorial. Then, given a lesser value
of the same number, this function calls itself. This continues until we reach the basic
case, in which there are no more function calls.
Recursion is a technique for handling complicated issues when the outcome
is dependent on the outcomes of smaller instances of the same issue.
If we think about functions, a function is said to be recursive if it keeps calling itself
until it reaches the base case.
Any recursive function has two primary components: the base case and the
recursive step. We stop going to the recursive phase once we reach the basic case.
To prevent endless recursion, base cases must be properly defined and are crucial.
The definition of infinite recursion is a recursion that never reaches the base case. If
a program never reaches the base case, stack overflow will continue to occur.

Recursion Types
Generally speaking, there are two different forms of recursion:
1. Linear Recursion
2. Tree Recursion

Linear Recursion
 A function that calls itself just once each time it executes is said to be linearly
recursive. A nice illustration of linear recursion is the factorial function. The name
"linear recursion" refers to the fact that a linearly recursive function takes a linear
amount of time to execute.

Take a look at the pseudo-code below:

function doSomething(n) {
// base case to stop recursion
if nis 0:
return
// here is some instructions
// recursive step
doSomething(n-1);
}
If we look at the function doSomething(n), it accepts a parameter named n and
does some calculations before calling the same procedure once more but with lower
values.
When the method doSomething() is called with the argument value n, let's say
that T(n) represents the total amount of time needed to complete the computation.
For this, we can also formulate a recurrence relation, T(n) = T(n-1) +
K. K serves as a constant here. Constant K is included because it takes time for the
function to allocate or de-allocate memory to a variable or perform a mathematical
operation. We use K to define the time since it is so minute and insignificant.
This recursive program's time complexity may be simply calculated since, in
the worst scenario, the method doSomething() is called n times. Formally speaking,
the function's temporal complexity is O(N).

Tree Recursion
When you make a recursive call in your recursive case more than once, it is
referred to as tree recursion. An effective illustration of Tree recursion is the
fibonacci sequence. Tree recursive functions operate in exponential time; they are
not linear in their temporal complexity.

Take a look at the pseudo-code below,

function doSomething(n) {
// base case to stop recursion
if n is less than 2:
return n;
// here is some instructions
// recursive step
return doSomething(n-1) + doSomething(n-2);
}
The only difference between this code and the previous one is that this one
makes one more call to the same function with a lower value of n.
Let's put T(n) = T(n-1) + T(n-2) + k as the recurrence relation for this function.
K serves as a constant once more.
When more than one call to the same function with smaller values is
performed, this sort of recursion is known as tree recursion. The intriguing aspect
is now: how time-consuming is this function?
Take a guess based on the recursion tree below for the same function.

DAA Recursion Tree Method

It may occur to you that it is challenging to estimate the time complexity by
looking directly at a recursive function, particularly when it is a tree recursion.
Recursion Tree Method is one of several techniques for calculating the temporal
complexity of such functions. Let's examine it in further detail.

What Is Recursion Tree Method?

Recurrence relations like T(N) = T(N/2) + N or the two we covered earlier in
the kinds of recursion section are solved using the recursion tree approach. These
recurrence relations often use a divide and conquer strategy to address problems.
It takes time to integrate the answers to the smaller sub problems that are created
when a larger problem is broken down into smaller sub problems.
The recurrence relation, for instance, is T(N) = 2 * T(N/2) + O(N) for the
Merge sort. The time needed to combine the answers to two sub problems with a
combined size of T(N/2) is O(N), which is true at the implementation level as well.
 For instance, since the recurrence relation for binary search is T(N) = T(N/2)
+ 1, we know that each iteration of binary search results in a search space that is cut
in half. Once the outcome is determined, we exit the function. The recurrence
relation has +1 added because this is a constant time operation.
The recurrence relation T(n) = 2T(n/2) + Kn is one to consider. Kn denotes
the amount of time required to combine the answers to n/2-dimensional sub
problems.
Let's depict the recursion tree for the aforementioned recurrence relation.

DAA Recursion Tree Method

We may draw a few conclusions from studying the recursion tree above,
including

1. The magnitude of the problem at each level is all that matters for determining the
value of a node. The issue size is n at level 0, n/2 at level 1, n/2 at level 2, and so on.

2. In general, we define the height of the tree as equal to log (n), where n is the size
of the issue, and the height of this recursion tree is equal to the number of levels in
the tree. This is true because, as we just established, the divide-and-conquer strategy
is used by recurrence relations to solve problems, and getting from issue size n to
problem size 1 simply requires taking log (n) steps.

Consider the value of N = 16, for instance. If we are permitted to divide N by

2 at each step, how many steps are required to get N = 1? Considering that we
are dividing by two at each step, the correct answer is 4, which is the value of log(16)
base 2.
log(16) base 2

log(2^4) base 2

4 * log(2) base 2, since log(a) base a = 1

so, 4 * log(2) base 2 = 4

3. At each level, the second term in the recurrence is regarded as the root.

Although the word "tree" appears in the name of this strategy, you don't need to be
an expert on trees to comprehend it.

How to Use a Recursion Tree to Solve Recurrence Relations?

The cost of the sub problem in the recursion tree technique is the amount of
time needed to solve the sub problem. Therefore, if you notice the phrase "cost"
linked with the recursion tree, it simply refers to the amount of time needed to solve
a certain sub problem.

Let's understand all of these steps with a few examples.

Example

Consider the recurrence relation,

T(n) = 2T(n/2) + K

Solution

The given recurrence relation shows the following properties,

A problem size n is divided into two sub-problems each of size n/2. The cost of
combining the solutions to these sub-problems is K.

Each problem size of n/2 is divided into two sub-problems each of size n/4 and so
on.
At the last level, the sub-problem size will be reduced to 1. In other words, we finally
hit the base case.

Let's follow the steps to solve this recurrence relation,

Step 1: Draw the Recursion Tree

DAA Recursion Tree Method

Step 2: Calculate the Height of the Tree

Since we know that when we continuously divide a number by 2, there comes a time
when this number is reduced to 1. Same as with the problem size N, suppose after K
divisions by 2, N becomes equal to 1, which implies, (n / 2^k) = 1

Here n / 2^k is the problem size at the last level and it is always equal to 1.

Now we can easily calculate the value of k from the above expression by taking log()
to both sides. Below is a more clear derivation,

n = 2^k

log(n) = log(2^k)
log(n) = k * log(2)
k = log(n) / log(2)
k = log(n) base 2
So the height of the tree is log (n) base 2.

Step 3: Calculate the cost at each level

Cost at Level-0 = K, two sub-problems are merged.

Cost at Level-1 = K + K = 2*K, two sub-problems are merged two times.
Cost at Level-2 = K + K + K + K = 4*K, two sub-problems are merged four times.
and so on....
Step 4: Calculate the number of nodes at each level

Let's first determine the number of nodes in the last level. From the recursion tree,
we can deduce this

Level-0 have 1 (2^0) node

Level-1 have 2 (2^1) nodes
Level-2 have 4 (2^2) nodes
Level-3 have 8 (2^3) nodes
So the level log(n) should have 2^(log(n)) nodes i.e. n nodes.

Step 5: Sum up the cost of all the levels

The total cost can be written as,

Total Cost = Cost of all levels except last level + Cost of last level
Total Cost = Cost for level-0 + Cost for level-1 + Cost for level-2 +.... + Cost for
level-log(n) + Cost for last level
The cost of the last level is calculated separately because it is the base case and no
merging is done at the last level so, the cost to solve a single problem at this level is
some constant value. Let's take it as O (1).

Let's put the values into the formulae,

T(n) = K + 2*K + 4*K + ..... + log(n)` times + `O(1) * n

T(n) = K(1 + 2 + 4 + ..... + log(n) times)` + `O(n)
T(n) = K(2^0 + 2^1 + 2^2 + .... + log(n) times + O(n)
If you closely take a look to the above expression, it forms a Geometric progression
(a, ar, ar^2, ar^3 ...... infinite time). The sum of GP is given by S(N) = a / (r - 1).
Here is the first term and r is the common ratio.
1. Lower Bound Theory:
According to the lower bound theory, for a lower bound L(n) of an algorithm,
it is not possible to have any other algorithm (for a common problem) whose time
complexity is less than L(n) for random input. Also, every algorithm must take at
least L(n) time in the worst case. Note that L(n) here is the minimum of all the
possible algorithms, of maximum complexity.

The Lower Bound is very important for any algorithm. Once we calculated it,
then we can compare it with the actual complexity of the algorithm and if their order
is the same then we can declare our algorithm as optimal. So in this section, we will
be discussing techniques for finding the lower bound of an algorithm.

Note that our main motive is to get an optimal algorithm, which is the one
having its Upper Bound the Same as its Lower Bound (U(n)=L(n)). Merge Sort is a
common example of an optimal algorithm.

Trivial Lower Bound –

It is the easiest method to find the lower bound. The Lower bounds which can
be easily observed based on the number of input taken and the number of output
produced are called Trivial Lower Bound.

Example: Multiplication of n x n matrix, where,

Input: For 2 matrices we will have 2n2 inputs

Output: 1 matrix of order n x n, i.e., n2 outputs
In the above example, it’s easily predictable that the lower bound is O(n2).

Computational Model –
The method is for all those algorithms that are comparison-based. For
example, in sorting, we have to compare the elements of the list among themselves
and then sort them accordingly. Similar is the case with searching and thus we can
implement the same in this case. Now we will look at some examples to understand
its usage.

Ordered Searching –
It is a type of searching in which the list is already sorted.
Example-1: Linear search
Explanation –
 In linear search, we compare the key with the first element if it does not match
we compare it with the second element, and so on till we check against the nth
element. Else we will end up with a failure.

Example-2: Binary search

Explanation –
In binary search, we check the middle element against the key, if it is greater
we search the first half else we check the second half and repeat the same process.
The diagram below there is an illustration of binary search in an array consisting of
4 elements

Calculating the lower bound: The max no of comparisons is n. Let there be k

levels in the tree.
No. of nodes will be 2k-1
The upper bound of no of nodes in any comparison-based search of an element
in the list of size n will be n as there are a maximum of n comparisons in worst case
scenario 2k-1
Each level will take 1 comparison thus no. of comparisons k≥|log2n|
Thus the lower bound of any comparison-based search from a list of n
elements cannot be less than log(n). Therefore we can say that Binary Search is
optimal as its complexity is Θ(log n).

Sorting –
The diagram below is an example of a tree formed in sorting combinations
with 3 elements.
Example – For n elements, finding lower bound using computation model.

Explanation –
For n elements, we have a total of n! combinations (leaf nodes). (Refer to the
diagram the total combinations are 3! or 6) also, it is clear that the tree formed is a
binary tree. Each level in the diagram indicates a comparison. Let there be k levels
=> 2k is the total number of leaf nodes in a full binary tree thus in this case we have
n!≤2k.

As the k in the above example is the no of comparisons thus by computational model

lower bound = k.

Now we can say that,

n!≤2T(n)
Thus,
T(n)>|log n!|
=> n!<=nn
Thus,
log n!<=log nn
Taking ceiling function on both sides, we get
|-log nn-|>=|-log n!-|
Thus complexity becomes Θ(lognn) or Θ(nlogn)

Using Lower bond theory to solve the algebraic problem:

Straight Line Program –

 The type of program built without any loops or control structures is called
the Straight Line Program. For example,

#include <iostream>

// Function to sum two numbers without using loops or control structures

int Sum(int a, int b) {
int c = a + b;
return c;
}

int main() {
// Example usage
int num1 = 5;
int num2 = 7;

int result = Sum(num1, num2);

std::cout << "The sum of " << num1 << " and " << num2 << " is: " << result <<
std::endl;

return 0;
}

Output
The sum of 5 and 7 is: 12

Algebraic Problem –
Problems related to algebra like solving equations inequalities etc. come under
algebraic problems. For example, solving equation ax2+bx+c with simple
programming.

#include <iostream>

int Algo_Sol(int a, int b, int c, int x) {

// 1 assignment
int v = a * x;

// 1 assignment
v = v + b;
 // 1 assignment
v = v * x;

// 1 assignment
int ans = v + c;
return ans;
}

int main() {
// Example usage
int result = Algo_Sol(2, 3, 4, 5);
std::cout << "Result: " << result << std::endl;

return 0;
}

Output
Result: 69

The complexity for solving here is 4 (excluding the returning).

The above example shows us a simple way to solve an equation for a
2-degree polynomial i.e., 4 thus for nth degree polynomial we will have a complexity
of O(n2).

Let us demonstrate via an algorithm.

Example: x+a0 is a polynomial of degree n.

#include <iostream>
#include <vector> // Include vector header for using vectors

// Function to calculate power of x raised to n

int power(int x, int n) {
int p = 1;

// Loop from 1 to n
for (int i = 1; i <= n; ++i) {
p *= x;
 }

return p;
}

// Function to evaluate the polynomial with coefficients A, value x, and degree n

int polynomial(std::vector<int>& A, int x, int n) {
int v = 0;

for (int i = 0; i <= n; ++i) {

// Loop within a loop from 0 to n
v += A[i] * power(x, i);
}

return v;
}

int main() {
// Example usage:
std::vector<int> A = {2, 3, 4}; // Coefficients of the polynomial
int x = 5; // Value of x
int n = A.size() - 1; // Degree of the polynomial

int result = polynomial(A, x, n);

std::cout << "Result: " << result << std::endl;

return 0;
}
//This ocde is contributed by Utkarsh.

Output
Result: 117

Loop within a loop => complexity = O(n2);

Now to find an optimal algorithm we need to find the lower bound here (as
per lower bound theory). As per Lower Bound Theory, The optimal algorithm to
solve the above problem is the one having complexity O(n). Let’s prove this theorem
using lower bounds.
Theorem: To prove that the optimal algo of solving a n degree polynomial is O(n)
Proof: The best solution for reducing the algo is to make this problem less complex
by dividing the polynomial into several straight-line problems.

=> anxn+an-1xn-1+an-2xn-2+...+a1x+a0
can be written as
((..(anx+an-1)x+..+a2)x+a1)x+a0
Now, the algorithm will be as,
v=0
v=v+an
v=v*x
v=v+an-1
v=v*x
...
v=v+a1
v=v*x
v=v+a0

#include <iostream>

int polynomial(int A[], int x, int n) {

int v = 0;

// loop executed n times

for (int i = n; i >= 0; i--) {
v = (v + A[i]) * x;
}

return v;
}

int main() {
// Example usage
int coefficients[] = {2, -1, 3}; // Coefficients of the polynomial 2x^2 - x + 3
int degree = 2; // Degree of the polynomial
int x_value = 4; // Value of x

int result = polynomial(coefficients, x_value, degree);

std::cout << "Result of the polynomial evaluation: " << result << std::endl;
 return 0;
}

Output
Result of the polynomial evaluation: 184

The complexity of this code is O(n). This way of solving such equations is
called Horner’s method. Here is where lower bound theory works and gives the
optimum algorithm’s complexity as O(n).

2. Upper Bound Theory:

According to the upper bound theory, for an upper bound U(n) of an
algorithm, we can always solve the problem at most U(n) time. Time taken by a
known algorithm to solve a problem with worse case input gives us the upper bound.
It’s difficult to provide a comprehensive list of advantages and disadvantages
of lower and upper bound theory, as it depends on the specific context in which it is
being used. However, here are some general advantages and disadvantages:

Advantages:
● Provides a clear understanding of the range of possible values for a quantity,
which can be useful in decision-making.
● Helps to identify the optimal value within the range of possible values,
which can lead to more efficient and effective solutions to problems.
● Can be used to prove the existence of solutions to optimization problems.
● Provides a theoretical framework for analyzing and solving a wide range of
mathematical problems.

Disadvantages:
● May not always provide a precise solution to optimization problems, as the
optimal value may not be within the range of possible values determined by
the lower and upper bounds.
● Can be computationally intensive, especially for complex optimization
problems with many constraints.
● May be limited by the accuracy of the data used to determine the lower and
upper bounds.
● Requires a strong mathematical background to use effectively.

Searching
Searching is the fundamental process of locating a specific element or item
within a collection of data. This collection of data can take various forms, such as
arrays, lists, trees, or other structured representations. The primary objective of
searching is to determine whether the desired element exists within the data, and if
so, to identify its precise location or retrieve it. It plays an important role in various
computational tasks and real-world applications, including information retrieval,
data analysis, decision-making processes, and more.

Searching terminologies:

Target Element:
In searching, there is always a specific target element or item that you want to
find within the data collection. This target could be a value, a record, a key, or any
other data entity of interest.

Search Space:
The search space refers to the entire collection of data within which you are
looking for the target element. Depending on the data structure used, the search
space may vary in size and organization.

Complexity:
Searching can have different levels of complexity depending on the data
structure and the algorithm used. The complexity is often measured in terms of time
and space requirements.

Deterministic vs. Non-deterministic:

Some searching algorithms, like binary search, are deterministic, meaning
they follow a clear, systematic approach. Others, such as linear search, are non-
deterministic, as they may need to examine the entire search space in the worst case.

Importance of Searching in DSA:

Efficiency: Efficient searching algorithms improve program performance.
Data Retrieval: Quickly find and retrieve specific data from large datasets.
Database Systems: Enables fast querying of databases.
Problem Solving: Used in a wide range of problem-solving tasks.

Applications of Searching:
Searching algorithms have numerous applications across various fields. Here
are some common applications:
1. Information Retrieval: Search engines like Google, Bing, and Yahoo use
sophisticated searching algorithms to retrieve relevant information from vast
amounts of data on the web.
2. Database Systems: Searching is fundamental in database systems for
retrieving specific data records based on user queries, improving efficiency in
data retrieval.
3. E-commerce: Searching is crucial in e-commerce platforms for users to find
products quickly based on their preferences, specifications, or keywords.
4. Networking: In networking, searching algorithms are used for routing packets
 efficiently through networks, finding optimal paths, and managing network
resources.
5. Artificial Intelligence: Searching algorithms play a vital role in AI
applications, such as problem-solving, game playing (e.g., chess), and
decision-making processes
6. Pattern Recognition: Searching algorithms are used in pattern matching tasks,
such as image recognition, speech recognition, and handwriting recognition.

Linear Search Algorithm:

Linear Search is a method for searching an element in a collection of elements.
In Linear Search, each element of the collection is visited one by one in a sequential
fashion to find the desired element. Linear Search is also known as Sequential
Search.

Algorithm:
Start: Begin at the first element of the collection of elements.
Compare: Compare the current element with the desired element.
Found: If the current element is equal to the desired element, return true or index to
the current element.
Move: Otherwise, move to the next element in the collection.
Repeat: Repeat steps 2-4 until we have reached the end of collection.
Not found: If the end of the collection is reached without finding the desired
element, return that the desired element is not in the array.

Working:
● Every element is considered as a potential match for the key and checked for
the same.
● If any element is found equal to the key, the search is successful and the
index of that element is returned.
● If no element is found equal to the key, the search yields “No match found”.

For example: Consider the array arr[] = {10, 50, 30, 70, 80, 20, 90, 40} and
key = 30

Step 1: Start from the first element (index 0) and compare key with each element
(arr[i]).

Comparing key with first element arr[0]. SInce not equal, the iterator moves to the
next element as a potential match.
Comparing key with next element arr[1]. SInce not equal, the iterator moves to the
next element as a potential match.

Step 2: Now when comparing arr[2] with key, the value matches. So the Linear
Search Algorithm will yield a successful message and return the index of the element
when key is found (here 2).

Time Complexity:
Best Case: In the best case, the key might be present at the first index. So the best
case complexity is O(1)
Worst Case: In the worst case, the key might be present at the last index i.e., opposite
to the end from which the search has started in the list. So the worst-case complexity
is O(N) where N is the size of the list.
Average Case: O(N)
Auxiliary Space: O(1) as except for the variable to iterate through the list, no other
variable is used.

Applications of Linear Search Algorithm:

 1. Unsorted Lists: When we have an unsorted array or list, linear search is most
commonly used to find any element in the collection.
2. Small Data Sets: Linear Search is preferred over binary search when we have
small data sets with
3. Searching Linked Lists: In linked list implementations, linear search is
commonly used to find elements within the list. Each node is checked
sequentially until the desired element is found.
4. Simple Implementation: Linear Search is much easier to understand and
implement as compared to Binary Search or Ternary Search.

Advantages of Linear Search Algorithm:

● Linear search can be used irrespective of whether the array is sorted or not.
It can be used on arrays of any data type.
● Does not require any additional memory.
● It is a well-suited algorithm for small datasets.

Disadvantages of Linear Search Algorithm:

● Linear search has a time complexity of O(N), which in turn makes it slow
for large datasets.
● Not suitable for large arrays.

When to use Linear Search Algorithm?

● When we are dealing with a small dataset.
● When you are searching for a dataset stored in contiguous memory.

Binary Search Algorithm:

Binary search is a search algorithm used to find the position of a target value
within a sorted array. It works by repeatedly dividing the search interval in half until
the target value is found or the interval is empty. The search interval is halved by
comparing the target element with the middle value of the search space.

Conditions to apply Binary Search Algorithm in a Data Structure:

● The data structure must be sorted.
● Access to any element of the data structure takes constant time.
 ● In this algorithm, Divide the search space into two halves by finding the
middle index “mid”.

● Compare the middle element of the search space with the key.
● If the key is found at middle element, the process is terminated.
● If the key is not found at middle element, choose which half will be used as
the next search space.
● If the key is smaller than the middle element, then the left side is used for
next search.
● If the key is larger than the middle element, then the right side is used for
next search.
● This process is continued until the key is found or the total search space is
exhausted.

To understand the working of binary search, consider the following

illustration:
Consider an array arr[] = {2, 5, 8, 12, 16, 23, 38, 56, 72, 91}, and the target = 23.

First Step: Calculate the mid and compare the mid element with the key. If the key
is less than mid element, move to left and if it is greater than the mid then move
search space to the right.

Key (i.e., 23) is greater than current mid element (i.e., 16). The search space moves
to the right.

Key is less than the current mid 56. The search space moves to the left.
If the key matches the value of the mid element, the element is found and stop search.

The Binary Search Algorithm can be implemented in the following two ways
1. Iterative Binary Search Algorithm
2. Recursive Binary Search Algorithm

Iterative Binary Search Algorithm:

Here we use a while loop to continue the process of comparing the key and
splitting the search space in two halves.

Implementation of Iterative Binary Search Algorithm

// C++ program to implement iterative Binary Search
#include <bits/stdc++.h>
using namespace std;

// An iterative binary search function.

int binarySearch(int arr[], int low, int high, int x)
{
while (low <= high) {
int mid = low + (high - low) / 2;

// Check if x is present at mid

if (arr[mid] == x)
return mid;

// If x greater, ignore left half

 if (arr[mid] < x)
low = mid + 1;

// If x is smaller, ignore right half

else
high = mid - 1;
}

// If we reach here, then element was not present

return -1;
}

// Driver code
int main(void)
{
int arr[] = { 2, 3, 4, 10, 40 };
int x = 10;
int n = sizeof(arr) / sizeof(arr[0]);
int result = binarySearch(arr, 0, n - 1, x);
(result == -1)
? cout << "Element is not present in array"
: cout << "Element is present at index " << result;
return 0;
}
Output
Element is present at index 3
Time Complexity: O(log N)
Auxiliary Space: O(1)

Recursive Binary Search Algorithm:

Create a recursive function and compare the mid of the search space with the
key. And based on the result either return the index where the key is found or call
the recursive function for the next search space.

Implementation of Recursive Binary Search Algorithm:

#include <bits/stdc++.h>
using namespace std;

// A recursive binary search function. It returns

// location of x in given array arr[low..high] is present,
// otherwise -1
int binarySearch(int arr[], int low, int high, int x)
{
 if (high >= low) {
int mid = low + (high - low) / 2;

// If the element is present at the middle

// itself
if (arr[mid] == x)
return mid;

// If element is smaller than mid, then

// it can only be present in left subarray
if (arr[mid] > x)
return binarySearch(arr, low, mid - 1, x);

// Else the element can only be present

// in right subarray
return binarySearch(arr, mid + 1, high, x);
}
}

// Driver code
int main()
{
int arr[] = { 2, 3, 4, 10, 40 };
int query = 10;
int n = sizeof(arr) / sizeof(arr[0]);
int result = binarySearch(arr, 0, n - 1, query);
(result == -1)
? cout << "Element is not present in array"
: cout << "Element is present at index " << result;
return 0;
}

Output
Element is present at index 3
Complexity Analysis of Binary Search Algorithm:
Time Complexity:
Best Case: O(1)
Average Case: O(log N)
Worst Case: O(log N)
Auxiliary Space: O(1), If the recursive call stack is considered then the auxiliary
space will be O(logN).
Applications of Binary Search Algorithm:
● Binary search can be used as a building block for more complex algorithms
used in machine learning, such as algorithms for training neural networks or
finding the optimal hyperparameters for a model.
● It can be used for searching in computer graphics such as algorithms for ray
tracing or texture mapping.
● It can be used for searching a database.

Advantages of Binary Search:

● Binary search is faster than linear search, especially for large arrays.
● More efficient than other searching algorithms with a similar time
complexity, such as interpolation search or exponential search.
● Binary search is well-suited for searching large datasets that are stored in
external memory, such as on a hard drive or in the cloud.

Disadvantages of Binary Search:

● The array should be sorted.
● Binary search requires that the data structure being searched be stored in
contiguous memory locations.
● Binary search requires that the elements of the array be comparable,
meaning that they must be able to be ordered.

Pattern Searching:
Pattern searching involves searching for a specific pattern or sequence of
elements within a given data structure. This technique is commonly used in string
matching algorithms to find occurrences of a particular pattern within a text or a
larger string. By using various algorithms like the Knuth-Morris-Pratt (KMP)
algorithm or the Rabin-Karp algorithm, pattern searching plays a crucial role in tasks
such as text processing, data retrieval, and computational biology.
Naive algorithm for Pattern Searching
Given text string with length n and a pattern with length m, the task is to
prints all occurrences of pattern in text.
Note: You may assume that n > m.

Examples:

Input: text = “THIS IS A TEST TEXT”, pattern = “TEST”

Output: Pattern found at index 10

Input: text = “AABAACAADAABAABA”, pattern = “AABA”

Output: Pattern found at index 0, Pattern found at index 9, Pattern found at index
12
Slide the pattern over text one by one and check for a match. If a match is found,
then slide by 1 again to check for subsequent matches.
#include <iostream>
#include <string>
using namespace std;

void search(string& pat, string& txt) {

int M = pat.size();
int N = txt.size();

// A loop to slide pat[] one by one

for (int i = 0; i <= N - M; i++) {
int j;
// For current index i, check for pattern match
for (j = 0; j < M; j++) {
if (txt[i + j] != pat[j]) {
break;
}
}

// If pattern matches at index i

if (j == M) {
cout << "Pattern found at index " << i << endl;
}
}
}

// Driver's Code
int main() {
// Example 1
string txt1 = "AABAACAADAABAABA";
string pat1 = "AABA";
cout << "Example 1: " << endl;
search(pat1, txt1);

// Example 2
string txt2 = "agd";
string pat2 = "g";
cout << "\nExample 2: " << endl;
search(pat2, txt2);

return 0;
}

Output
Pattern found at index 0
Pattern found at index 9
Pattern found at index 13
Time Complexity: O(N2)
Auxiliary Space: O(1)

Complexity Analysis of Naive algorithm for Pattern Searching:

Best Case: O(n)
 When the pattern is found at the very beginning of the text (or very early
on).
The algorithm will perform a constant number of comparisons, typically on
the order of O(n) comparisons, where n is the length of the pattern.
Worst Case: O(n2)
When the pattern doesn’t appear in the text at all or appears only at the very
end.
The algorithm will perform O((n-m+1)*m) comparisons, where n is the
length of the text and m is the length of the pattern.
In the worst case, for each position in the text, the algorithm may need to
compare the entire pattern against the text.

KMP Algorithm for Pattern Searching

Given a text txt[0 . . . N-1] and a pattern pat[0 . . . M-1], write a function
search(char pat[], char txt[]) that prints all occurrences of pat[] in txt[]. You may
assume that N > M.

Examples:

Input: txt[] = “THIS IS A TEST TEXT”, pat[] = “TEST”

Output: Pattern found at index 10

Input: txt[] = “AABAACAADAABAABA”

pat[] = “AABA”
Output: Pattern found at index 0, Pattern found at index 9, Pattern found at index
12
The worst case complexity of the Naive algorithm is O(m(n-m+1)). The time
complexity of the KMP algorithm is O(n+m) in the worst case.

KMP (Knuth Morris Pratt) Pattern Searching:

The Naive pattern-searching algorithm doesn’t work well in cases where we

see many matching characters followed by a mismatching character.

Examples:

1) txt[] = “AAAAAAAAAAAAAAAAAB”, pat[] = “AAAAB”

2) txt[] = “ABABABCABABABCABABABC”, pat[] = “ABABAC” (not a worst
case, but a bad case for Naive)

The KMP matching algorithm uses degenerating property (pattern having the
same sub-patterns appearing more than once in the pattern) of the pattern and
improves the worst-case complexity to O(n+m).

The basic idea behind KMP’s algorithm is: whenever we detect a mismatch
(after some matches), we already know some of the characters in the text of the next
window. We take advantage of this information to avoid matching the characters
that we know will anyway match.
Matching Overview

txt = “AAAAABAAABA”
pat = “AAAA”
We compare first window of txt with pat

txt = “AAAAABAAABA”
pat = “AAAA” [Initial position]
We find a match. This is same as Naive String Matching.

In the next step, we compare next window of txt with pat.

txt = “AAAAABAAABA”
pat = “AAAA” [Pattern shifted one position]

This is where KMP does optimization over Naive. In this second window, we
only compare fourth A of pattern with fourth character of current window of text to
decide whether current window matches or not. Since we know first three characters
will anyway match, we skipped matching first three characters.

Need of Preprocessing?
An important question arises from the above explanation, how to know how
many characters to be skipped. To know this, we pre-process pattern and prepare an
integer array lps[] that tells us the count of characters to be skipped

Preprocessing Overview:
KMP algorithm preprocesses pat[] and constructs an auxiliary lps[] of size m
(same as the size of the pattern) which is used to skip characters while matching.
Name lps indicates the longest proper prefix which is also a suffix. A proper
prefix is a prefix with a whole string not allowed. For example, prefixes of “ABC”
are “”, “A”, “AB” and “ABC”. Proper prefixes are “”, “A” and “AB”. Suffixes of
the string are “”, “C”, “BC”, and “ABC”.
We search for lps in subpatterns. More clearly we focus on sub-strings of
patterns that are both prefix and suffix.
For each sub-pattern pat[0..i] where i = 0 to m-1, lps[i] stores the length of the
maximum matching proper prefix which is also a suffix of the sub-pattern pat[0..i].
lps[i] = the longest proper prefix of pat[0..i] which is also a suffix of pat[0..i].
Note: lps[i] could also be defined as the longest prefix which is also a proper suffix.
We need to use it properly in one place to make sure that the whole substring is not
considered.

Examples of lps[] construction:

For the pattern “AAAA”, lps[] is [0, 1, 2, 3]

For the pattern “ABCDE”, lps[] is [0, 0, 0, 0, 0]

For the pattern “AABAACAABAA”, lps[] is [0, 1, 0, 1, 2, 0, 1, 2, 3, 4, 5]

For the pattern “AAACAAAAAC”, lps[] is [0, 1, 2, 0, 1, 2, 3, 3, 3, 4]

For the pattern “AAABAAA”, lps[] is [0, 1, 2, 0, 1, 2, 3]

Preprocessing Algorithm:
In the preprocessing part, We calculate values in lps[]. To do that, we keep
track of the length of the longest prefix suffix value (we use len variable for this
purpose) for the previous index
We initialize lps[0] and len as 0.
If pat[len] and pat[i] match, we increment len by 1 and assign the incremented
value to lps[i].
If pat[i] and pat[len] do not match and len is not 0, we update len to lps[len-
1]

Illustration of preprocessing (or construction of lps[]):

pat[] = “AAACAAAA”

=> len = 0, i = 0:

lps[0] is always 0, we move to i = 1

=> len = 0, i = 1:

Since pat[len] and pat[i] match, do len++,

store it in lps[i] and do i++.
Set len = 1, lps[1] = 1, i = 2
=> len = 1, i = 2:
Since pat[len] and pat[i] match, do len++,
store it in lps[i] and do i++.
Set len = 2, lps[2] = 2, i = 3
=> len = 2, i = 3:

Since pat[len] and pat[i] do not match, and len > 0,

Set len = lps[len-1] = lps[1] = 1
=> len = 1, i = 3:

Since pat[len] and pat[i] do not match and len > 0,

len = lps[len-1] = lps[0] = 0
=> len = 0, i = 3:

Since pat[len] and pat[i] do not match and len = 0,

Set lps[3] = 0 and i = 4
=> len = 0, i = 4:

Since pat[len] and pat[i] match, do len++,

Store it in lps[i] and do i++.
Set len = 1, lps[4] = 1, i = 5
=> len = 1, i = 5:

Since pat[len] and pat[i] match, do len++,

Store it in lps[i] and do i++.
Set len = 2, lps[5] = 2, i = 6
=> len = 2, i = 6:

Since pat[len] and pat[i] match, do len++,

Store it in lps[i] and do i++.
len = 3, lps[6] = 3, i = 7
=> len = 3, i = 7:

Since pat[len] and pat[i] do not match and len > 0,

Set len = lps[len-1] = lps[2] = 2
=> len = 2, i = 7:

Since pat[len] and pat[i] match, do len++,

Store it in lps[i] and do i++.
len = 3, lps[7] = 3, i = 8
We stop here as we have constructed the whole lps[].
Implementation of KMP algorithm:
Unlike the Naive algorithm, where we slide the pattern by one and compare
all characters at each shift, we use a value from lps[] to decide the next characters to
be matched. The idea is to not match a character that we know will anyway match.

How to use lps[] to decide the next positions (or to know the number of characters
to be skipped)?

● We start the comparison of pat[j] with j = 0 with characters of the current

window of text.
● We keep matching characters txt[i] and pat[j] and keep incrementing i and j
while pat[j] and txt[i] keep matching.
● When we see a mismatch
● We know that characters pat[0..j-1] match with txt[i-j…i-1] (Note that j
starts with 0 and increments it only when there is a match).
● We also know (from the above definition) that lps[j-1] is the count of
characters of pat[0…j-1] that are both proper prefix and suffix.
From the above two points, we can conclude that we do not need to match
these lps[j-1] characters with txt[i-j…i-1] because we know that these characters will
anyway match. Let us consider the above example to understand this.

Below is the illustration of the above algorithm:

Consider txt[] = “AAAAABAAABA“, pat[] = “AAAA“

If we follow the above LPS building process then lps[] = {0, 1, 2, 3}

-> i = 0, j = 0: txt[i] and pat[j] match, do i++, j++

-> i = 1, j = 1: txt[i] and pat[j] match, do i++, j++

-> i = 2, j = 2: txt[i] and pat[j] match, do i++, j++

-> i = 3, j = 3: txt[i] and pat[j] match, do i++, j++

-> i = 4, j = 4: Since j = M, print pattern found and reset j, j = lps[j-1] = lps[3] = 3

Here unlike Naive algorithm, we do not match first three

characters of this window. Value of lps[j-1] (in above step) gave us index of next
character to match.

-> i = 4, j = 3: txt[i] and pat[j] match, do i++, j++

-> i = 5, j = 4: Since j == M, print pattern found and reset j, j = lps[j-1] = lps[3] = 3

Again unlike Naive algorithm, we do not match first three characters of this window.
Value of lps[j-1] (in above step) gave us index of next character to match.

-> i = 5, j = 3: txt[i] and pat[j] do NOT match and j > 0, change only j. j = lps[j-1]
= lps[2] = 2

-> i = 5, j = 2: txt[i] and pat[j] do NOT match and j > 0, change only j. j = lps[j-1]
= lps[1] = 1

-> i = 5, j = 1: txt[i] and pat[j] do NOT match and j > 0, change only j. j = lps[j-1]
= lps[0] = 0

-> i = 5, j = 0: txt[i] and pat[j] do NOT match and j is 0, we do i++.

-> i = 6, j = 0: txt[i] and pat[j] match, do i++ and j++

-> i = 7, j = 1: txt[i] and pat[j] match, do i++ and j++

We continue this way till there are sufficient characters in the text to be compared
with the characters in the pattern…

Below is the implementation of the above approach:

// C++ program for implementation of KMP pattern searching

// algorithm

#include <bits/stdc++.h>

void computeLPSArray(char* pat, int M, int* lps);

// Prints occurrences of pat[] in txt[]

void KMPSearch(char* pat, char* txt)
{
int M = strlen(pat);
 int N = strlen(txt);

// create lps[] that will hold the longest prefix suffix

// values for pattern
int lps[M];

// Preprocess the pattern (calculate lps[] array)

computeLPSArray(pat, M, lps);

int i = 0; // index for txt[]

int j = 0; // index for pat[]
while ((N - i) >= (M - j)) {
if (pat[j] == txt[i]) {
j++;
i++;
}

if (j == M) {
printf("Found pattern at index %d ", i - j);
j = lps[j - 1];
}

// mismatch after j matches

else if (i < N && pat[j] != txt[i]) {
// Do not match lps[0..lps[j-1]] characters,
// they will match anyway
if (j != 0)
j = lps[j - 1];
else
i = i + 1;
}
}
}

// Fills lps[] for given pattern pat[0..M-1]

void computeLPSArray(char* pat, int M, int* lps)
{
// length of the previous longest prefix suffix
int len = 0;
 lps[0] = 0; // lps[0] is always 0

// the loop calculates lps[i] for i = 1 to M-1

int i = 1;
while (i < M) {
if (pat[i] == pat[len]) {
len++;
lps[i] = len;
i++;
}
else // (pat[i] != pat[len])
{
// This is tricky. Consider the example.
// AAACAAAA and i = 7. The idea is similar
// to search step.
if (len != 0) {
len = lps[len - 1];

// Also, note that we do not increment

// i here
}
else // if (len == 0)
{
lps[i] = 0;
i++;
}
}
}
}

// Driver code
int main()
{
char txt[] = "ABABDABACDABABCABAB";
char pat[] = "ABABCABAB";
KMPSearch(pat, txt);
return 0;
}
Output
Found pattern at index 10
Time Complexity: O(N+M) where N is the length of the text and M is the length of
the pattern to be found.
Auxiliary Space: O(M)

Rabin-Karp Algorithm for Pattern Searching:

Given a text T[0. . .n-1] and a pattern P[0. . .m-1], write a function search(char
P[], char T[]) that prints all occurrences of P[] present in T[] using Rabin Karp
algorithm. You may assume that n > m.

Examples:

Input: T[] = “THIS IS A TEST TEXT”, P[] = “TEST”

Output: Pattern found at index 10

Input: T[] = “AABAACAADAABAABA”, P[] = “AABA”

Output: Pattern found at index 0
Pattern found at index 9
Pattern found at index 12

In the Naive String Matching algorithm, we check whether every substring of

the text of the pattern’s size is equal to the pattern or not one by one.

Like the Naive Algorithm, the Rabin-Karp algorithm also check every
substring. But unlike the Naive algorithm, the Rabin Karp algorithm matches the
hash value of the pattern with the hash value of the current substring of text, and if
the hash values match then only it starts matching individual characters. So Rabin
Karp algorithm needs to calculate hash values for the following strings.

● Pattern itself
● All the substrings of the text of length m which is the size of pattern.

How is Hash Value calculated in Rabin-Karp?

Hash value is used to efficiently check for potential matches between a pattern
and substrings of a larger text. The hash value is calculated using a rolling hash
function, which allows you to update the hash value for a new substring by
efficiently removing the contribution of the old character and adding the contribution
of the new character. This makes it possible to slide the pattern over the text and
calculate the hash value for each substring without recalculating the entire hash from
scratch.
Here’s how the hash value is typically calculated in Rabin-Karp:

Step 1: Choose a suitable base and a modulus:

Select a prime number ‘p‘ as the modulus. This choice helps avoid overflow issues
and ensures a good distribution of hash values.
Choose a base ‘b‘ (usually a prime number as well), which is often the size of the
character set (e.g., 256 for ASCII characters).
Step 2: Initialize the hash value:

Set an initial hash value ‘hash‘ to 0.

Step 3: Calculate the initial hash value for the pattern:

Iterate over each character in the pattern from left to right.

For each character ‘c’ at position ‘i’, calculate its contribution to the hash value as
‘c * (bpattern_length – i – 1) % p’ and add it to ‘hash‘.
This gives you the hash value for the entire pattern.
Step 4: Slide the pattern over the text:

Start by calculating the hash value for the first substring of the text that is the same
length as the pattern.
Step 5: Update the hash value for each subsequent substring:

To slide the pattern one position to the right, you remove the contribution of the
leftmost character and add the contribution of the new character on the right.
The formula for updating the hash value when moving from position ‘i’ to ‘i+1’ is:
hash = (hash - (text[i - pattern_length] * (bpattern_length - 1)) % p) * b + text[i]
Step 6: Compare hash values:

When the hash value of a substring in the text matches the hash value of the
pattern, it’s a potential match.
If the hash values match, we should perform a character-by-character comparison
to confirm the match, as hash collisions can occur.
Step-by-step approach:

1. Initially calculate the hash value of the pattern.

2. Start iterating from the starting of the string:
3. Calculate the hash value of the current substring having length m.
4. If the hash value of the current substring and the pattern are same check if
the substring is same as the pattern.
5. If they are same, store the starting index as a valid answer. Otherwise,
continue for the next substrings.
6. Return the starting indices as the required answer.

Below is the implementation of the above approach:

/* Following program is a C++ implementation of Rabin Karp

Algorithm given in the CLRS book */
#include <bits/stdc++.h>
using namespace std;

// d is the number of characters in the input alphabet

#define d 256

/* pat -> pattern

txt -> text
q -> A prime number
*/
void search(char pat[], char txt[], int q)
{
int M = strlen(pat);
int N = strlen(txt);
int i, j;
int p = 0; // hash value for pattern
int t = 0; // hash value for txt
int h = 1;

// The value of h would be "pow(d, M-1)%q"

for (i = 0; i < M - 1; i++)
h = (h * d) % q;

// Calculate the hash value of pattern and first

// window of text
for (i = 0; i < M; i++) {
p = (d * p + pat[i]) % q;
t = (d * t + txt[i]) % q;
}

// Slide the pattern over text one by one

for (i = 0; i <= N - M; i++) {

// Check the hash values of current window of text

// and pattern. If the hash values match then only
// check for characters one by one
if (p == t) {
/* Check for characters one by one */
for (j = 0; j < M; j++) {
if (txt[i + j] != pat[j]) {
break;
}
}

// if p == t and pat[0...M-1] = txt[i, i+1,

// ...i+M-1]

if (j == M)
cout << "Pattern found at index " << i
<< endl;
}
 // Calculate hash value for next window of text:
// Remove leading digit, add trailing digit
if (i < N - M) {
t = (d * (t - txt[i] * h) + txt[i + M]) % q;

// We might get negative value of t, converting

// it to positive
if (t < 0)
t = (t + q);
}
}
}

/* Driver code */
int main()
{
char txt[] = "GEEKS FOR GEEKS";
char pat[] = "GEEK";

// we mod to avoid overflowing of value but we should

// take as big q as possible to avoid the collison
int q = INT_MAX;

// Function Call
search(pat, txt, q);
return 0;
}

// This is code is contributed by rathbhupendra

Output
Pattern found at index 0
Pattern found at index 10
Time Complexity:

The average and best-case running time of the Rabin-Karp algorithm is O(n+m), but
its worst-case time is O(nm).
The worst case of the Rabin-Karp algorithm occurs when all characters of pattern
and text are the same as the hash values of all the substrings of T[] match with the
hash value of P[].
Auxiliary Space: O(1)

Limitations of Rabin-Karp Algorithm

Spurious Hit: When the hash value of the pattern matches with the hash value
of a window of the text but the window is not the actual pattern then it is called a
spurious hit. Spurious hit increases the time complexity of the algorithm. In order to
minimize spurious hit, we use good hash function. It greatly reduces the spurious hit.