22AIM42 -DAA-Module 3: The Greedy Method 1

MODULE 3
THE GREEDY METHOD

1. THE GENERAL METHOD:

 The greedy method is a straightforward design technique, wherein the problems have n
inputs and require us to obtain a subset that satisfies some constraints.
 Any subset that satisfies these constraints is called a feasible solution
 We need to find a feasible solution that either maximizes or minimizes a given objective
function. Such a feasible solution is called as an optimal solution.
 The greedy method says that algorithm design to take place in stages, considering one
input at a time.
 At each stage, a decision is made regarding whether a particular input is in an
optimal solution and is done by some selection procedure.
Selection procedure itself is based on some optimization measure. This measure
may be the objective function.
 If the inclusion of the next input into the partially constructed optimal solution
results in an infeasible solution, then this input is not added to the partial solution.
O.w., it is added.
 Using selection procedure, consisting of optimization measures results in an
algorithm that generate suboptimal solutions. This version of the greedy technique
is called the subset paradigm.
 The control abstraction for the subset paradigm is shown below:

 For some problems, wherein the selection of an optimal subset is not made, the greedy
method works by making the decisions by considering the inputs in some order. Here,
each decision is made using an optimization criterion that can be computed using
decision already made. This version of greedy method is called ordering paradigm.

 Ex: [ Machine scheduling]: we are given n tasks and an infinite supply of machines on
which these tasks can be performed.
Each task has a start time si and a finish time fi , si<fi. [si,fi] is the processing interval for
task i. Two tasks i and j overlap iff their processing intervals overlap at a point other than
the interval start or end. For ex., the interval [1,4] overlaps with [2,4] but not with[5,8].
A feasible task: Machine assignment, in which no machine is assigned two
overlapping tasks. Therefore, each machine works on atmost one task at any time.
Optimal assignment: It is a feasible assignment that utilizes the fewest number of
machines.
Suppose n=7 tasks , labeled a through g and their start and finish times are shown
below:
task a b c d e f g
start 0 3 4 9 7 1 6
finish 2 7 7 11 10 5 8

Dept. of ICBS SJEC

22AIM42 -DAA-Module 3: The Greedy Method 2
Greedy approach:
 Assign the tasks in stages, one task per stage and in an increasing order of task start
times.
 Call a machine old if at least one task has been assigned to it. If a machine is not old,
it is new.
 For machine selection, use the criteria: if an old machine becomes available by the
start time of the task to be assigned , assign the task to this maschine; if not assign it to a
new machine.

The tasks in an increasing order of their start time are : a, f, b, c, g, e, d. in each stage, one
task is assigned to the machine in an order.
The final solution is depicted below:

c d
M3 ---------------- --------------

f g
M2

a b e
M1

0 1 2 3 4 5 6 7 8 9 10 11

Time

2. MINIMUM COST SPANNING TREES:

Let G=(V,E) be an undirected graph. A subgraph t=(V, E′ ) of G is a spanning tree of G iff t is
a tree.
Ex:

Spanning trees
The cost of a spanning tree is the sum of the costs of the edges in that tree. Therefore, a span-
ning tree of G with minimum cost forms minimum cost spanning tree.
Definition: A spanning tree of a connected graph is its connected acyclic subgraph (i.e., a
tree) that contains all the vertices of the graph. A minimum spanning tree of a weighted
connected graph is its spanning tree of the smallest weight, where the weight of a tree is de-
fined as the sum of the weights on all its edges. The minimum spanning tree problem is the
problem of finding a minimum spanning tree for a given weighted connected graph.

Dept. of ICBS SJEC

22AIM42 -DAA-Module 3: The Greedy Method 3

2.1 PRIM’S ALGORITHM:

 Prim's algorithm constructs a minimum spanning tree through a sequence of expanding
sub- trees. The initial subtree in such a sequence consists of a single vertex selected
arbitrarilyfrom the set V of the graph's vertices.

 On each iteration it expands the current tree in the greedy manner by simply attaching
to it the nearest vertex not in that tree. (By the nearest vertex, we mean a vertex not in
the tree connected to a vertex in the tree by an edge of the smallest weight. Ties can
be broken arbitrarily.)
 The algorithm stops after all the graph's vertices have been included in the tree being
constructed. Since the algorithm expands a tree by exactly one vertex on each of its it-
erations, the total number of such iterations is n - 1, where n is the number of vertices in
the graph. The tree generated by the algorithm is obtained as the set of edges used for
tree expansions.
 The pseudocode of this algorithm is shown below:

 Correctness: Prim’s algorithm always yields a minimum spanning tree

 Example: An example of prim’s algorithm is shown below.
The parenthesized labels of a vertex in the middle column indicate the nearest
tree vertex and edge weight; selected vertices and edges are shown in bold.
Consider the following graph.

Dept. of ICBS SJEC

22AIM42 -DAA-Module 3: The Greedy Method 4

Analysis of Efficiency
The efficiency of Prim’s algorithm depends on the data structures chosen for the graph itself
and for the priority queue of the set V − VT whose vertex priorities are the distances to the
nearest tree vertices.
1. If a graph is represented by its weight matrix and the priority queue is implemented
as an unordered array, the algorithm’s running time will be in Θ(|V|2). Indeed, on
each of the |V| − 1iterations, the array implementing the priority queue is traversed to
find and delete the minimum and then to update, if necessary, the priorities of the re-
maining vertices.
We can implement the priority queue as a min-heap. (A min-heap is a complete bi-
nary treein which every element is less than or equal to its children.) Deletion of the

Dept. of ICBS SJEC

22AIM42 -DAA-Module 3: The Greedy Method 5
smallest element from and insertion of a new element into a min-heap of size n are
O(log n) operations.
2. If a graph is represented by its adjacency lists and the priority queue is implemented
as a min-heap, the running time of the algorithm is in O(|E| log |V |).
This is because the algorithm performs |V| − 1 deletions of the smallest element and
makes |E| verifications and, possibly, changes of an element’s priority in a min-heap of
size not exceeding |V|. Each of these operations, as noted earlier, is a O(log |V|) opera-
tion. Hence, the running time of this implementation of Prim’s algorithm is in
(|V| − 1+ |E|) O (log |V |) = O(|E| log |V |) because, in a connected graph, |V| − 1≤ |E|.

2.2 KRUSKAL’S ALGORITHM:

 Background:
Kruskal's algorithm is another greedy algorithm for the minimum spanning tree prob-
lem thatalso always yields an optimal solution. It is named Kruskal's algorithm, after Jo-
seph Kruskal.
Kruskal's algorithm looks at a minimum spanning tree for a weighted connected
graph G =(V, E) as an acyclic sub graph with |V | - 1 edges for which the sum of the
edge weights is the smallest. Consequently, the algorithm constructs a minimum
spanning tree as an expanding sequence of sub graphs, which are always acyclic but
are not necessarily connected on the intermediate stages of the algorithm.
 Working: The algorithm begins by sorting the graph's edges in non decreasing order of
their weights. Then, starting with the empty sub graph, it scans this sorted list adding the
next edge on the list to the current sub graph if such an inclusion does not create a cycle
and simply skipping the edge otherwise.

The fact that ET ,the set of edges composing a minimum spanning tree of graph G actually a
tree in Prim's algorithm but generally just an acyclic sub graph in Kruskal's algorithm.
Kruskal’s algorithm is not simpler because it has to check whether the addition of the next
edge to the edges already selected would create a cycle.

Dept. of ICBS SJEC

22AIM42 -DAA-Module 3: The Greedy Method 6
We can consider the algorithm's operations as a progression through a series of forests
containing all the vertices of a given graph and some of its edges. The initial forest consists of
|V| trivial trees, each comprising a single vertex of the graph. The final forest consists of a
single tree, which is a minimum spanning tree of the graph. On each iteration, the
algorithm takes the next edge (u, v) from the sorted list of the graph's edges, finds the trees
containing the vertices u and v, and, if these trees are not the same, unites them in a larger
tree by adding the edge (u, v).

Analysis of Efficiency
The crucial check whether two vertices belong to the same tree can be found out using union-
find algorithms.
Efficiency of Kruskal’s algorithm is based on the time needed for sorting the edge weights
of a given graph. Hence, with an efficient sorting algorithm, the time efficiency of Kruskal's
algorithm will be in O (|E| log |E|).
Example: An illustration is shown here. The edges selected are shown in bold.
Consider the following graph.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 7

3. SINGLE SOURCE SHORTEST PATHS:

Single-source shortest-paths problem is defined as follows. For a given vertex called the
source in a weighted connected graph, the problem is to find shortest paths to all its other verti-
ces. The single-source shortest-paths problem asks for a family of paths, each leading from the
source to a different vertex in the graph, though some paths may, of course, have edges in
common.

3.1 Dijkstra's Algorithm

Dijkstra's Algorithm is the best-known algorithm for the single-source shortest-paths
problem. This algorithm is applicable to undirected and directed graphs with nonnegative
weights only.
Working - Dijkstra's algorithm finds the shortest paths to a graph's vertices in order of their
distance from a given source.
 First, it finds the shortest path from the source to a vertex nearest to it, then to a
second nearest, and so on.
 In general, before its ith iteration commences, the algorithm has already identi-
fied the shortest paths to i-1 other vertices nearest to the source. These vertices, the
source, and the edges of the shortest paths leading to them from the source form a subtree Ti
of the given graph shown in the figure.

 Since all the edge weights are nonnegative, the next vertex nearest to the source can
be found among the vertices adjacent to the vertices of Ti. The set of vertices adjacent

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 8
to the vertices in Ti can be referred to as "fringe vertices"; they are the candidates
from which Dijkstra's algorithm selects the next vertex nearest to the source.
 To identify the ith nearest vertex, the algorithm computes, for every fringe vertex u,
the sum of the distance to the nearest tree vertex v (given by the weight of the edge (v,
u)) and the length d., of the shortest path from the source to v (previously determined
by the algorithm) and then selects the vertex with the smallest such sum. The fact that
it suffices to compare the lengths of such special paths is the central insight of Dijks-
tra's algorithm.
 To facilitate the algorithm's operations, we label each vertex with two labels
o The numeric label d indicates the length of the shortest path from the source to
this vertex found by the algorithm so far; when a vertex is added to the tree, d
indicates the length of the shortest path from the source to that vertex.
o The other label indicates the name of the next-to-last vertex on such a path, i.e.,
the parent of the vertex in the tree being constructed. (It can be left unspeci-
fied for the source s a d vertices that are adjacent to none of the current tree vertices.)
With such labeling, finding the next nearest vertex u* becomes a simple task of
finding a fringe vertex with the smallest d value. Ties can be broken arbitrarily.
 After we have identified a vertex u* to be added to the tree, we need to perform two
operations:
o Move u* from the fringe to the set of tree vertices.
o For each remaining fringe vertex u that is connected to u* by an edge of
weight w (u*, u) such that d u*+ w(u*, u) <d u, update the labels of u by u*
and du* + w(u*, u), respectively
 . The pseudocode of Dijkstra’s algorithm is given below. Note that in the
following pseudocode, VT contains a given source vertex and the fringe contains the
vertices adjacent to it after iteration 0 is completed.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 9
Illustration: An example of Dijkstra's algorithm is shown below. The next closest
vertex is shown in bold

The shortest paths (identified by following

nonnumeric labels backward from a destina-
tion vertex in the left column to the source)
and their lengths (given by numeric labels of
the tree vertices) are as follows->:

Analysis:
The time efficiency of Dijkstra’s algorithm depends on the data structures used for
implementing the priority queue and for representing an input graph itself. For graphs
represented by their adjacency lists and the priority queue implemented as a min-heap, it is in
O ( |E| log |V| )
Applications
 Transportation planning and packet routing in communication networks, including the
Internet
 Finding shortest paths in social networks, speech recognition, document formatting,
robotics, compilers, and airline crew scheduling.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 10
4. OPTIMAL TREE PROBLEM

Background
Suppose we have to encode a text that comprises characters from some n-character alphabet
by assigning to each of the text's characters some sequence of bits called the codeword. There
are two types of encoding: Fix ed-length encoding, Variable-length encoding
Fixed-length encoding: This method assigns to each character a bit string of the same length
m (m >= log2 n). This is exactly what the standard ASCII code does. One way of getting a
coding scheme that yields a shorter bit string on the average is based on the old idea of as-
signing shorter code-words to more frequent characters and longer code-words to less fre-
quent characters.
Variable-length encoding: This method assigns code-words of different lengths to different
characters, introduces a problem that fixed-length encoding does not have. Namely, tell how
many bits of an encoded text represent the first (or, more generally, the i th ) symbol? To avoid
this complication, we can limit ourselves to the so-called prefix-free (or simply prefix) codes.
In a prefix code, no codeword is a prefix of a codeword of another symbol. Hence, with such an
encoding, we can simply scan a bit string until we get the first group of bits that is a codeword
for some symbol, replace these bits by this symbol, and repeat this operation until the bit
string’s end is reached.
If we want to create a binary prefix code for some alphabet, it is natural to associate the al-
phabet's characters with leaves of a binary tree in which all the left edges are labelled by 0 and
all the right edges are labelled by 1 (or vice versa). The codeword of a character can then be
obtained by recording the labels on the simple path from the root to the character's leaf. Since
there is no simple path to a leaf that continues to another leaf, no codeword can be a prefix of
another codeword; hence, any such tree yields a prefix code.
Among the many trees that can be constructed in this manner for a given alphabet with known
frequencies of the symbol occurrences, how can we construct a tree that would assign short-
er bit strings to high-frequency symbols and longer ones to low-frequency symbols? It can
be done by the following greedy algorithm, invented by David Huffman

4.1 HUFFMAN TREES AND CODES

HUFFMAN’S ALGORITHM

Step 1 Initialize n one-node trees and label them with the symbols of the alphabet given.
Record the frequency of each symbol in its tree’s root to indicate the tree’s weight. (More
generally, the weight of a tree will be equal to the sum of the frequencies in the tree’s
leaves.)
Step 2 Repeat the following operation until a single tree is obtained. Find two trees with the
smallest weight (ties can be broken arbitrarily,). Make them the left and right subtree of a
new tree and record the sum of their weights in the root of the new tree as its weight.
A tree constructed by the above algorithm is called a Huffman tree. It define in the
manner described above—a Huffman code.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 11

EXAMPLE Consider the five-symbol alphabet {A, B, C, D, _} with the following occurrence
frequencies in a text made up of these symbols:

The Huffman tree construction for this input is shown in below:

The resulting codewords are as follows:

Hence, DAD is encoded as 011101, and 10011011011101 is decoded as BAD_AD.

With the occurrence frequencies given and the codeword lengths obtained, the average
number of bits per symbol in this code is
2* 0.35 + 3* 0.1 + 2 *0.2 + 2* 0.2 + 3*0.15 = 2.25.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 12
Had we used a fixed-length encoding for the same alphabet, we would have to use at least 3
bits per each symbol. Thus, for this example, Huffman’s code achieves the compression ratio
(a standard measure of a compression algorithm’s effectiveness) of (3−2.25)/3*100%= 25%.
In other words, Huffman’s encoding of the above text will use 25% less memory than its
fixed-length encoding.

5. SPACE AND TIME TRADE OFFS

The idea is to preprocess the problem’s input, in whole or in part, and store the additional in-
formation obtained to accelerate solving the problem afterward. We call this approach input
enhancement. (Eg: Sorting by Counting)
The other type of technique that exploits space-for-time trade-offs simply uses extra space to fa-
cilitate faster and/or more flexible access to the data. We call this approach pre-structuring.
(Eg: BTrees)

5.1 Sorting by Counting:

The idea is to count, for each element of a list to be sorted, the total number of elements smaller
than this element and record the results in a table. These numbers will indicate the positions of
the elements in the sorted list: e.g., if the count is 10 for some element, it should be in the 11th
position (with index 10, if we start counting with 0) in the sorted array.
Thus, we will be able to sort the list by simply copying its elements to their appropriate posi-
tions in a new, sorted list. This algorithm is called comparison counting sort.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 13
Analysis of Efficiency:
The number of times its basic operation, the comparison A[i]<A[j ], is executed is equal to the
sum:

Advantages:
 The algorithm makes the same number of key comparisons as selection sort and in addi-
tion uses a linear amount of extra space.
 The algorithm makes the minimum number of key moves possible, placing each of them
directly in their final position in a sorted array.
 The counting idea does work productively in a situation in which elements to be sorted
belong to a known small set of values. Assume, for example, that we have to sort a list
whose values can be either 1 or 2. Rather than applying a general sorting algorithm, we
should be able to take advantage of this additional information about values to be sort-
ed.

Distribution Counting: - In this approach, the elements of A whose values are equal to the
lowest possible value l are copied into the first F[0]elements of S, i.e., positions 0 through
F[0]− 1; the elements of value l + 1 are copied to positions from F[0] to (F[0]+ F[1]) − 1; and
so on. Since such accumulated sums of frequencies are called a distribution in statistics, the
method itself is known as distribution counting.

Example: Consider Sorting the array

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 14

Analysis of Efficiency:
Assuming that the range of array values is fixed, this is a linear algorithm because it makes just
two consecutive passes through its input array A. This is a better time-efficiency class than that
of the most efficient sorting algorithms—mergesort, quicksort, and heapsort.
It is important to remember, however, that this efficiency is obtained by exploiting the specific
nature of inputs for which sorting by distribution counting works, in addition to trading space
for time.

5.2 B-Trees

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 15

Analysis of efficiency:
Searching in a B-tree is very similar to searching in the binary search tree, Starting with the
root, we follow a chain of pointers to the leaf that may contain the search key. Then we search
for the search key among the keys of that leaf. Since keys are stored in sorted order, at both pa-
rental nodes and leaves, we can use binary search if the number of keys at a node is large
enough to make it worthwhile.
When used for storing a large data file on a disk, the nodes of a B-tree normally correspond to
the disk pages. Since the time needed to access a disk page is typically several orders of magni-
tude larger than the time needed to compare keys in the fast computer memory, it is the number
of disk accesses that becomes the principal indicator of the efficiency of this and similar data
structures.
How many nodes of a B-tree do we need to access during a search for a record with a given
key value? This number is, obviously, equal to the height of the tree plus 1.
For any B-tree of order m with n nodes and height h>0, we have the following inequality:

After a series of standard simplifications, this inequality reduces to

which, in turn, yields the following upper bound on the height h of the B-tree of order m with n
nodes:

Thus, it implies that searching in a B-tree is a O(log n) operation.

The algorithm for inserting a new record into a Btree is outlined below:
First, we apply the search procedure to the new record’s key K to find the appropriate leaf for
the new record.
If there is room for the record in that leaf, we place it there (in an appropriate position so
that the keys remain sorted) and we are done.
If there is no room for the record, the leaf is split in half by sending the second half of the
records to a new node.

Dept. of ICBS SJEC

22AIM42-DAA-Module 3: The Greedy Method 16
After that, the smallest key K in the new node and the pointer to it are inserted into the old
leaf’s parent (immediately after the key and pointer to the old leaf).
This recursive procedure may percolate up to the tree’s root. If the root is already full too, a new
root is created with the two halves of the old root’s keys split between two children of the new
root. As an example, Figure 7.9 shows the result of inserting 65 into the B-tree in Figure 7.8 un-
der the restriction that the leaves cannot contain more than three items.

****************************************************************************

Dept. of ICBS SJEC