HASH FUNCTION h: h(key) = POSITION in which to place the record in the table.

This is a function, that uses the key of the record, to calculate the position in which to
place the record in the table.

COLLISION: The hash function may produce the same result for several different keys.
Collisions are then said to occur. These will be resolved a little later on.

Open addressing, or closed hashing, is a method of collision resolution in hash tables.

With this method a hash collision is resolved by probing, or searching through alternate
locations in the array (the probe sequence) until either the target record is found, or an
unused array slot is found, which indicates that there is no such key in the table. [1] Well
known probe sequences include:

Linear probing
in which the interval between probes is fixed — often at 1.
Quadratic probing
in which the interval between probes increases linearly (hence, the indices are
described by a quadratic function).
Double hashing
in which the interval between probes is fixed for each record but is computed by
another hash function.

Hash table. Open addressing strategy

Chaining is a good way to resolve collisions, but it has additional memory cost to store
the structure of linked-lists. If entries are small (for instance integers) or there are no
values at all (set ADT), then memory waste is comparable to the size of data itself. When
hash table is based on the open addressing strategy, all key-value pairs are stored in the
hash table itself and there is no need for external data structure.

Collision resolution

Let's consider insertion operation. If the slot, key is hashed to, turns out to be busy
algorithm starts seeking for a free bucket. It starts with the hashed-to slot and proceeds in
a probe sequence, until free bucket is found. There are several well known probe
sequences:
  linear probing: distance between probes is constant (i.e. 1, when probe examines
consequent slots);
 quadratic probing: distance between probes increases by certain constant at each
step (in this case distance to the first slot depends on step number quadratically);
 double hashing: distance between probes is calculated using another hash
function.

Open addressing strategy requires, that hash function has additional properties. In
addition to performing uniform distribution, it should also avoid clustering of hash
values, which are consequent in probe's order.

Collisions

Life is not perfect with Hashing. The Hashing function can produce the same position to
place two or more records. Clearly this is unacceptable. The simplest solution when this
happens is just to try the next position in the table & so on until an empty slot is found.

Lets continue the example:

Insert: 420 Joe 1342

h = (20) Mod 11 = 9

We attempt to insert this record into position 9. We cant because it is full.

Add 1 to the position i.e. position 10 & try there. Fine it is empty so insert the record
there (Note I am omitting the Name & Tel No from the description).

ID
0
1
2 46
3
4 81
5
6 39
7
8
 ID
9 64
10 20
Position

Now insert: 31 Zoe 1328

h = (31) Mod 11 = 9

We attempt to insert this record into position 9. We cant because it is full.

Add 1 to the position i.e. position 10 & try there. No good it is full.

Add 1 to the position i.e. position 11. Well position 11 does not exist. It is off the end of
the table. So try position 0 at the start of the table instead. It is empty so insert the record.
there. (If it was full you would try position 1 & so on until you find an empty position).

The mathematical description of going beyond the end of the table & wrapping around to
the start to the table can be simply written as Mod (Table-size) or Mod 11 in this case.

The final table is:

ID
0 20
1
2 46
3
4 81
5
6 39
7
8
9 64
10 20
Position

The advantages of this method are:

1. The method is simple to implement; and

 2. As long as there is an empty slot in the table a new record can be inserted.

The disadvantages of the method are:

1. It can be inefficient. It often occurs that a large blocks of fully occupied cells form
inside the table. This is called Primary Clustering. When a new record is hashed
to some value in this full cluster it takes many attempts before an empty slot can
be found. This is illustrated in the next example.
2. If the table is 50% full then it can be shown that an empty slot can be found after
2.5 attempts. If the table is 90% full then some 50 attempts are required to find an
empty slot.

3. To keep the method efficient we must keep the table at most half full. Once this
happens a new table must be set up. Its size needs to be the 1st prime number that
is at least double the old table size. Then we must re-hash all the records in the old
table into the new table (This is a lot of work). Then we can carry on.

4. Deletion problem. This occurs when a record is deleted from the table and this
record is on the path to some other record that was subsequently inserted. The
search hits an empty slot, stops and reports that the record is not there. In fact it is
but the method can not find it. The solution to this problem is to mark each
position as EMPTY, FULL or DELETED. When we a searching for a record if
the position is marked as DELETED the search continues. The search only stops
when the record is found or a EMPTY slot is found. This is illustrated in the next
example.

Hashing with open addressing - linear probing

What I have described is Hashing with open addressing with collisions resolved by
Linear probing.

This can be written in a general form as:

Hash = (h(x) + f(i) i=0,1,2…) Mod (Table-size)

In the next example:

h(x) = key Mod 11, the Table-size = 11 and f(i) = i, giving

Hash = ( key Mod 11+ i i=0,1,2…) Mod 11

For illustration purposes 5 records are hashed into the table. These records have the keys:
51, 62, 74, 19 ,73
Key Hash

51 7 OK
62 7 => (7 + 1) = 8 OK
74 8 => (8 + 1) = 9 OK
19 8 => 9 =>10 OK
73 7 => 8 => 9 => 10 =>11 = 0 OK

Empty After 51 After 62 After 74 After 19 After 73

0 73
1
2
3
4
5
6
7 51 51 51 51 51
8 62 62 62 62
9 74 74 74
10 19 19

Now lets FIND record 73. We use exactly the same procedure as inserting the record.

Key Hash

73 7 Is the key in position 7 = 73

(51<>73) No Try next position

8 Is the key in position 8 = 73

(62<>73) No Try next position

9 Is the key in position 9 = 73

(74<>73) No Try next position

10 Is the key in position 10 = 73

(19<>73) No Try next position

0 Is the key in position 0 = 73

(67<>73) Yes Found

Right this example WORKS.

Now : DELETE record 74. The resulting table is:

Empty After 51 After 62 After 74 After 19 After 73 After 74 Deleted

0 73 73
1
2
3
4
5
6
7 51 51 51 51 51 51
8 62 62 62 62 62
9 74 74 74
10 19 19 19

Lets try to FIND record 73. We use exactly the same procedure as inserting the record.

Key Hash

73 7 Is the key in position 7 = 73

(51<>73) No Try next position
8 Is the key in position 8 = 73
(62<>73) No Try next position
9 Record EMPTY thus stop search
=> record NOT found. This is incorrect.

WE needed to mark this record as DELETED so the search can continue on to position
10 then 0 where the record is found.

Hashing with open addressing - quadratic probing

The next strategy is to use f(i) = i²

We try 1² =1 position away from the original hash, then 2² = 4 positions away, then 3² = 9
positions away & so on.

Hash = (h(x) + f(i) i=0,1,2…) Mod (Table-size)

In the next example: h(x) = key Mod 11, the Table-size = 11 and f(i) = i². Which gives;

Hash = ( key Mod 11 + i² i=0,1,2…) Mod 11

For illustration purposes 6 records are hashed into the table. These records have the keys:
51, 62, 74, 19, 73

Key Hash

51 7 OK

62 7 => (7 + 1²) = 8 OK

74 8 => (8 + 1²) = 9 OK

19 8 => (8 + 1²) = 9 => (8 + 2²) = 12 Mod(11) = 1 OK

73 7 => (7 + 1²) = 8 => (7 + 2²) = 11 Mod(11) = 0

OK

84 7 => (7 + 1²) = 8 => (7 + 2²) = 11 Mod(11) = 0

=> (7 + 3²) = 16 Mod(11) =5 OK

95 7 => (7 + 1²) = 8 => (7 + 2²) = 11 Mod(11) = 0

=> (7 + 3²) = 16 Mod(11) =5
=> (7 + 4²) = 23 Mod(11) = 1
=> (7 + 5²) = 32 Mod(11) = 10 OK

106 7 Now try it yourself – you will discover that while there
are still empty slots in the table the hash function wont
find them – POTENTIAL DISASTER! Don’t worry there is a
solution.

Empty After 51 After 62 After 7 After 19 After 73 After 77 After 84 After 95

0 73 73 73 73
1 19 19 19 19 19
2
3
4
5 84 84
6
7 51 51 51 51 51 51 51 51
8 62 62 62 62 626 62 62
9 74 74 74 74 74 74
10 95
Advantages of the method:

1. Primary clustering is reduced & effectively eliminated.

Disadvantages of the method:

1. the calculation to do the hash is a bit more complex;

2. the table can still have empty slots but the hash function & collision resolution
does not find them. This problem can be solved in a simple fashion. The table size
must be a prime number and the table MUST be kept less than half full. When
these two conditions are met it can be proved that a record will be placed in the
table. Once this happens a new table must be set up. Its size needs to be the 1st
prime number that is at least double the old table size. Then we must re-hash all
the records in the old table into the new table (This is a lot of work). Then we can
carry on inserting further values.

3. While quadratic probing is better than linear probing it still has a disadvantage.
Every record that hashes to the same position ( 62, 73, 84 ,95 ) all follow exactly
the same path away from the original position occupied by 51. As more records t
hash to this position the longer the collision resolution process takes to find an
empty slot. This is known as secondary clustering.

Hashing with open addressing - double hashing

The next strategy is to use f(i) = i * h2(x)

When we have a collision we use a second hash function to determine the distance away
from the first position to try to placethe record, if that fails we move that distance again &
so on i.e. we try at:

h1(x)

h1(x) + h2(x)

h1(x) + 2 * h2(x)

h1(x) + 3 * h2(x); and so on

Hash = (h(x) + f(i) i=0,1,2…) Mod (Table-size) becomes Hash = ( h1(x) + i * h2(x)
i=0,1,2…) Mod (Table-size)

To ensure that this method words the following conditions must hold:
 1. h2(x) must NOT be ZERO ( If it was ever zero then not other positions would ever
be tried)
2. All cells must be capable of being tried. The following formula for h2(x) is a good
one:

h2(x) = R – (x mod R) where R is a prime number less than the Table-size

In the next example: h1(x) = key Mod 11, the Table-size = 11 and h2(x) = 7 – (key mod 7)

The following 6 records with keys: 38 , 1, 16 ,49, 11, 60 will be hashed to the table.

Key h1(x)
h2(x) = 7 – (key mod 7)
h1(x) + i * h2(x) i=1,2,3..

38 38 Mod 11 = 5 OK

1 1 Mod 11 = 1 OK

16 16 Mod 11 = 5 full
7-16 Mod 7=7–2=5
Hash=5+5=10 OK

49 49 Mod 11 = 5 full
7-49 Mod 7=7–0=7
Hash=5+7=12 Mod 11 = 1 full
Hash=5+14=19 Mod 11 = 8 OK

11 11 Mod 11 = 0 OK

60 60 Mod 11 = 5 full
7-60 Mod 7=7-4=3
Hash=5+3=8 full
Hash=5+2*3=5+6=11Mod 11=0 full
Hash=5+3*3=5+9=14Mod 1 =3 OK

Empty After 38 AFter 1 After 16 After 49 After 11 After 60

0 11 11
1 1 1 1 1 1
2
3 60
4
5 38 38 38 38 38 38
 Empty After 38 AFter 1 After 16 After 49 After 11 After 60
6
7
8 49 49 49 49
9
10 16 16 16 16 16

Advantages of the method: Records with ID 16, 49, 60 all hashed to position 5 which
clashed with where record 38 had already been placed.In the case of 16 a position 5 away
was tried & was successful.In the case of 49 a position 7 away then 14 away was tried. In
the case of 60 a position 3, 6,then 9 away was tried until successful. In quadratic hashing
all clashes to a position follow exactly the same path away from this position. In double
hashing DIFFERENT paths away from the clash position are followed. This reduces the
length of path that has to be traversed before an empty slot is found. Naturally not every
clash follows a different path. Some will follow the same path as others.

Disadvantages of the method: the calculation to do the second hash is a bit more
complex.

Rehashing

Rehashing is the process that needs to be done when the table of entries gets too full.
Normally one rehashes once the table becomes 50% full. The reason for this is that a)
when linear probing is used the number of attempts required to find an empty slot
increases dramatically as the table fills; and b) with quadratic probing the method is only
guaranteed to work is the table is less than 50% filled.

The process of re-hashing is simple but time consuming.

1. A new table is created that is at least double the size of the original table. The
usual choice for the new table size is the first prime number that is at least twice
as big as the old table size. In the example below the old table size was 5. The
new table size is 11. 11 is the first prime that is at least double the old table size of
5.

2. Go through all the entries in the old table and rehash each one to the new table. In
this example the old hash was h1 = key Mod 5 for the table of size 5. For the new
table of size 11 the new hash would be h2 = key Mod 11.

Example: Old hash: h1 = key Mod 5

Insert 24, 14, 34
key h1 = key Mod 5
24 24 Mod 5 = 4
14 14 Mod 5 = 4 full, try 5 Mod 5 = 0 OK
34 34 Mod 5 = 4 full, try 5 Mod 5 = 0 full, try 0 + 1 = 1 OK

But the table is now more that 50% full with 3 entries in a 5 slot table.

Initially After 24 inserted After 14 inserted After 34 inserted

0 14 14
1 34
2
3
4 24 24 24

Create a new 11 slot table. Start at the top of the old table and re-hash each entry using h2
= key Mod 11

key h2 = key Mod 11

1414 Mod 11 = 3OK
3434 Mod 11 = 1OK
2424 Mod 11 = 2OK

As you can see the records hash to different positions in the new 11 slot table.

Initially After 14 inserted After 34 inserted After 24 inserted

0
1 34 34
2 24
3 14 14 14
4
5
6
7
8
9
10
Separate Chaining

A final method of resolving collisions is to use a table with one entry for every possible
hash value. When a collision occurs a linked list is created with the new record being
inserted at the end of that particular list.

Hash h = Key Mod 11 for a table-size of 11.

Example: Hash h = Key Mod 11

KeyHash
22 Joe0
26 Pam4
39 Zulu6
48 Henk4
17 Bob6
The disadvantage of the method is:
 1. Space has to be allocated for the pointer within each record.

2. It can take considerable time to find the required record.

The advantages of the method are:

1. The Table-size is constant. The lists from each entry grow as entries are inserted
into the table.
2. The number of entries in the table usually exceed the table size. This is because
several values collide and are put onto a list.

Review Questions

Suggested answer at the end of the chapter.

1. Explain what is ment by “hashing”.

2. What is ment by a collision?

3. What is primary clustering & why does it occur?

4. What is secondary clustering & why does it occur?

5. What are the advantages & disadvantages of using a long key?

6. What is the advantage of using double hashing as a collision resolution method?

Problems

Suggested answer at the end of the chapter.

1. Insert the following records into a table of size 17 using the hash function h = key
mod 17. Linear probing is to be used to resolve collisions. The keys of the records
are: 2, 5, 14, 31, 15, 32, 48, 49.
2. Insert the following records into a table of size 13 using the hash function h = key
mod 13. Quadratic probing is to be used to resolve collisions. The keys of the
records are: 1, 2, 5, 14, 10, 23
 3. Insert the following records into a table of size 13 using the hash function h = key
mod 13. Double hashing is to be used to resolve collisions. The second hash
function is : h2 = 1 + (key Mod 5). The keys of the records are: 1, 24, 8, 10, 12,
21, 47, 34

Answers

Answers to Review Questions

1. HASHING is a method that uses a function and the key of the record to find a
position to place the record in a table { h(key) = POSITION }. The record must be
inserted, found or removed in CONSTANT time. Generally the range of the keys
is much greater than the size of the table.
2. COLLISION: The hash function may produce the same result for several different
keys. A collisions is then said to occur.

3. PRIMARY CLUSTERING can occur when linear probing is used to resolve

collisions. This means that the next available position after the original one is
chosen to place the record. If many records collide they are d as close as possible
to the original position. This causes many cells close together to all be filled.

4. SECONDARY CLUSTERING occurs when quadratic probing is used to resolve

collisions. Exactly the same path away from the original position is used (1, 4, 9,
16, 25, 36, etc) until an empty slot is found. This exact path is tried until an empty
slot is found. The more collisions there are the longer this path becomes.

5. LONG KEYS

ADVANTAGES : Give a good distribution of keys over the entire table. This
means that the records are evenly distributed over the entire table.

DISADVANTAGES: It takes longer to calculate the hash function. Overflow

problems can occur.

6. DOUBLE HASHING. In quadratic hashing all clashes to a position follow

exactly the same path away from this position. In double hashing DIFFERENT
paths away from the clash position are followed. This reduces the length of path
that has to be traversed before an empty slot is found. Naturally not every clash
follows a different path. Some will follow the same path as others.

Answers to Problems
1. Key h = key mod 17
2. 2 2 Mod 17 = 2 OK
3. 5 5 Mod 17 = 5 OK
4. 14 14 Mod 17 = 14 OK
5. 31 31 Mod 17 = 14 =>15 OK
6. 15 15 Mod 17 = 15=>16 OK
7. 32 32 Mod 17 = 16=>17 = 0 OK
8. 48 48 Mod 17 = 14=>15=>16=>17 = 0=>1 OK
9. 49 49 Mod 17 = 15=>16=>17 = 0=>1=>2 =>3 OK
10.
11.
12. 1, 2, 5,14, 10, 23
13.
14. Key h = key Mod 13
15. 1 1 Mod 13 = 1 OK
16. 2 2 Mod 13 = 2 OK
17. 5 5 Mod 13 = 5 OK
18. 14 14 Mod 13 = 1 Full,
19. 1 + 1² = 2 Full,
20. 1 + 2²=5 Full,
21. 1 + 3²=10 OK
22. 10 10 Mod 13 = 10 Full,
23. 10 + 1² = 11 OK
24. 23 23 Mod 13 = 10 Full,
25. 10 + 1² = 11 Full,
26. 10 + 2²=14 Mod 13 = 1 Full,
27. 10 + 3²=19 Mod 13 = 6 OK
28.
29. h = key mod 13.
30. h2 = 1+(key Mod 5).
31. The keys of the records are: 1, 24, 8, 10, 12, 21, 47, 34
32.
33. Key key Mod 13 1 + (key Mod 5).
34.
35. 1 1 Mod 13 = 1 OK
36. 24 24 Mod 13 = 11 OK
37. 8 8 Mod 13 = 8 OK
38. 10 10 Mod 13 = 10 OK
39. 12 12 Mod 13 = 12 OK
40. 1 1 Mod 13 = 1 OK
41. 21 21 Mod 13 = 8 Full 1 + 21 Mod 5 = 1 + 1 = 2
42. 8 + 2 = 10 Full
43. 8 + 2 * 2 = 12 Full
44. 8 + 3 * 2 = 14 Mod 13 = 1 Full
45. 8 + 4 * 2 = 16 Mod 13 = 3 OK
46. 47 47 Mod 13 = 8 Full 1 + 47 Mod 5 = 1 + 2 = 3
47. 8 + 3 = 11 Full
48. 8 + 2 * 3 = 14 Mod 13 = 1 Full
 49. 8 + 3 * 3 = 17 Mod 13 = 4 OK
50. 34 34 Mod 13 = 8 Full 1 + 34 Mod 5 = 1 + 4 = 5
51. 8 + 5 = 13 Mod 13 = 0 OK
52.

Priority Queues

A priority queue is a queue of records. Each record will contain, in its data, a field that is
used as the criterion of importance to order that item relative to all the other items. Items
arrive in any order. They exit the queue in minimum order.

Example: A file arrives to be printed. The arrival time is recorded. Files are removed
from the queue in arrival-time order. The example below implies that the records are kept
in a list which is ordered on arrival time. This certainly is a possibility but it is not the
optimal one. A different structure, which is more efficient, will be described in this
section.

Arrivals Priority Queue (Symbolic)

File 1 9h25 File 1, 0925
File 2 10h27 File 1, 0925; File 2 1027
File 3 6h23 File 3 623; File 1, 0925; File 2 1027
File 4 10h05 File 3 623; File 1, 0925; File 4 1005; File 2 1027

A PRIORITY QUEUE should be thought of as an Abstract Data Structure with the two
methods INSERT and DELETEMIN:

 Insert places a new item in the priority queue in the correct place.
 DeleteMin removes the smallest item then the Priority queue is reordered

Example of Priority Queues

 Printer queue where the priority of a job could be the importance of the person
submitting it. It could also be the arrival time of the job. It could also be the
combination of these two criteria.
 Scheduler for an Operating System

 Queue in a shortest path graph algorithm

 A queue for Discrete Event Simulation

 Sorting method using Heapsort

Priority Queue implemented as a Binary Heap

A HEAP is a special BINARY TREE. It is a COMPLETE binary tree except possibly for
the lowest (or leaf) level. This leaf level is filled strictly from left to right as illustrated
below. (If another node is added it would be added as the right child of node “E”.

A heap has 2 special properties:

1. The STRUCTURE property

2. The HEAP ORDER property

A Binary Heap requires no pointers in its implementation and is optimally represented as

an array.
Structure Property

The STRUCTURE PROPERTY is simply that the Heap is a COMPLETELY filled

binary tree with the possible exception of the bottom level. The bottom level is filled
from left to right. Using this array data structure there are the following useful & efficient
properties.

1. For any node at position I: Except for the root. The parent of the node I is in
position I div 2

Examples:

Node J is at position 10 = I - Parent position is I div 2 = 10 div 2 = position 5.

This is Node E.

Node G is at position 7 = I - Parent position is I div 2 = 7 div 2 = position 3. This

is Node C (Remember with “div” any remainder is omitted.)

2. For any element in the heap in position I,

 Its left child will be found at position 2*I,

 Its right child will be found at position 2*I + 1

Examples:

For Node A at position 1,

 Its left child Node B is at position 2*I = 2*1 =2

 Its right child Node C is at position 2*I + 1 = 2*1 + 1 = 3

For Node D at position 4

 Its left child Node B is at position 2*I = 2*4 = 8

 Its right child Node C is at position 2*I + 1 = 2*4 + 1 = 9

These two properties make it trivially easy and efficient to do these computations.
It should be appreciated that when you are at some node in the heap you either
wish to get access to the parent or to one of the children.

3. Height of the Heap.There are N nodes and it is a complete binary tree. Thus the
height of the tree is at most log2 N (actually .Lower bound[log2 N]). This is a
consequence of complete tree of height h having between 2h and 2h+1 – 1 nodes.

Example 1:

H 2H 2H + 1 – 1 .
 0 20 = 1 2 0+1 – 1 = 2 – 1 = 1 This is just the root node with ht = 0
1 21 = 2 21+1 – 1 = 4 – 1 = 3 With ht = 1 you have 2 or 3 nodes
2 22 = 4 22+1 – 1 = 8 – 1 = 7 With ht = 2 you have 4 to 7 nodes
3 23 = 8 23+1 – 1 = 16 – 1 = 15 With ht = 3 you have 8 to 15 nodes

The importance of the height of the tree is that: we can place any new element in
the heap in an appropriate place by at most 1calculation at each level of the tree.
This means that the work done is at most the height of the tree which is log2 N.
This will be demonstrated in the following sections.

Heap Order Property

For every node in the heap, except the root, the key of the parent <= key of the child

Below is an example of a heap and a non-heap:

Basic Heap Operations

1. Insert and item in a Priority Queue

The algorigthm is as follows:

a. Insert the new item at the next position in the leaf (or lowest) layer of the
heap. Assume its value is “X”. If the lowest layer is completely full start a
new layer & insert the item in th e first position of that layer.
b. If “X” is at the root of the tree: Quit. The heap is in order, compare “X”
with the value of is parent.

 If “X” >= value of parent: Quit. The heap is in order.

  If “X” < value of parent: Swop these values.

REPEAT step 2 (starting at the new position of “X”).

This is known as a “percolate up” strategy.

 Example 1

Insert the item with value 17 into the heap.The initial state of the
unordered ‘heap’ is shown below.

Compare the 17 with the value of it parent which is 38.

Swop these values.

This gives the following ‘heap’.

Compare the 17 with the value of it parent which is 35.

Swop these values.

This gives the following ‘heap’.

 Compare the 17 with the value of it parent which is 12.

It is less than the value of its parent.

The heap is in order. QUIT.

 Example 2

Instead of inserting a record with value 17 we had inserted a record with

value 7 then: The same sequence would occur.
Compare the 7 with the value of it parent which is 12.

Swop these values.

This gives the following ‘heap’

 7 is at the root of the tree: Quit. The heap is in order.

The heap has a maximum height of : lower bound ( log2 N ) = log2 11 =~

log2 23.3 = 3. As you can see you require at most 3 operations to put the
new record in the correct place. Often you need less operations than this.

2. Delete the Minimum Value from the Heap

To delete the minimum value from the heap is very easy. This value is in the root
node. It is simply removed from there.The heap is now not in order. So it must put
back into its correct order.
 The algorithm is:

a. The value of the root node has been removed. Consider this to be the
‘hole’ node.
b. Remove the right-most node on the lowest level & keep it in “temp”

c. Find the smallest child node of the ‘hole’ node. If it has no children: put
‘temp’ in ‘hole” node; Quit

d. Compare the ‘temp” node with this smallest child node

e. If “temp” node >= smallest child node:

i. Put smallest child record in hole record

ii. Change ‘hole’ node to smallest child node

iii. Go to step 3

b. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node

ii. Quit

 Example 1
a. The value of the root node has been removed. Consider this to be the ‘hole’ node.
b. Remove the right-most node on the lowest level & keep it in “temp”temp = 38

c. Find the smallest child node of the ‘hole’ node. Its node with value = 17. If it has
no children: put ‘temp’ in ‘hole” node; Quit

d. Compare the ‘temp” node with this smallest child node

e. If “temp” node >= smallest child node:

i. Put smallest child record in hole record. Put 17 in 'hole'

node
 ii. Change ‘hole’ node to smallest child node. 'hole' is old
node 17

iii. Go to step 3. Go to step 3

f. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node

ii. Quit

g. The value of the root node has been removed. Consider this to be the ‘hole’ node.
h. Remove the right-most node on the lowest level & keep it in “temp”

i. Find the smallest child node of the ‘hole’ node. Its node with value = 35. If it has
no children: put ‘temp’ in ‘hole” node; Quit

j. Compare the ‘temp” node with this smallest child node

k. If “temp” node >= smallest child node:

i. Put smallest child record in hole record. Put 35 in 'hole'

node

ii. Change ‘hole’ node to smallest child node. 'hole' is old

node 35

iii. Go to step 3. Go to step 3

l. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node

ii. Quit
m. The value of the root node has been removed. Consider this to be the ‘hole’ node.
n. Remove the right-most node on the lowest level & keep it in “temp”

o. Find the smallest child node of the ‘hole’ node. Its node with value = 43. If it has
no children: put ‘temp’ in ‘hole” node; Quit

p. Compare the ‘temp” node with this smallest child node

q. If “temp” node >= smallest child node:

i. Put smallest child record in hole record.

 ii. Change ‘hole’ node to smallest child node.

iii. Go to step 3.

r. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node. Put 38 in 'hole' node

ii. Quit

 Example 2
 Had the value 38 in the original tree been 58 instead the process would
have continued on for one further iteration before it was placed. The
remaining steps are illustrated below.

a. The value of the root node has been removed. Consider this to be the ‘hole’ node.
b. Remove the right-most node on the lowest level & keep it in “temp”

c. Find the smallest child node of the ‘hole’ node. Its node with value = 43. If it has
no children: put ‘temp’ in ‘hole” node; Quit

d. Compare the ‘temp” node with this smallest child node

e. If “temp” node >= smallest child node:

i. Put smallest child record in hole record. Put 43 in 'hole'

node

ii. Change ‘hole’ node to smallest child node. 'hole' is old

node 43

iii. Go to step 3. Go to step 3

f. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node.

ii. Quit
g. The value of the root node has been removed. Consider this to be the ‘hole’ node.
h. Remove the right-most node on the lowest level & keep it in “temp”

i. Find the smallest child node of the ‘hole’ node. If it has no children: put ‘temp’ in
‘hole” node; Quit Put 58 in 'hole' node. Quit

j. Compare the ‘temp” node with this smallest child node

k. If “temp” node >= smallest child node:

i. Put smallest child record in hole record.

 ii. Change ‘hole’ node to smallest child node.

iii. Go to step 3.

l. If “temp” node < smallest child node:

i. Put “temp” node in ‘hole’ node.

ii. Quit

Heap Sort

As the name implies Heapsort is an algorithm that can be used to sort records into order.
Naturally it uses a heap to effect the sort.

The algorithm is as follows:

1. Insert all the N records in a heap;

2. While heap not empty deleteMin.

This is a very efficient algorithm:

 To insert N records takes time: O(N * log2N)

 To delete & reorder N items takes O ( N *(1 + log2N) )

 In total the order of the sort is O(N * log2N)

At first glance one may think that the space required for the sort will be:

 N elements for the unsorted array; and

 N elements for the sorted array.

By a clever, but simple, trick only one array of N elements is needed. Essentially the
unsorted array is kept in the left hand side of the array while the sorted array is kept in the
right hand side of the array.

Example: Use heapsort to sort the following records into ascending order

1. Insert all the values into the heap. The picture below shows the original heap after
all the values have been inserted.
2. Remove the minimum element. Keep this value however. The new reduced heap
is shown below.
3. Observe that the 7th slot in the array is empty. Store the removed element there.
The sorted elements are marked. In other words the element that has been
removed from the array is stored in the slot that has been freed up by the heap
shrinking by one element. The array thus becomes:

4. Now remove the minimum element & keep it. Reordeer the heap. Put the
minimum element into the 6th slot. This is the slot that is no longer required by
the decreased heap.
5. Now remove the minimum element & keep it. Reordeer the heap. Put the
minimum element into the 5th slot. This is the slot that is no longer required by
the decreased heap.
 6. Continue this process until all the elements are in order. This is shown below.

If you want the records sorted in ascending order then access them from the last element
of the array to the first. If you want then in descending order access them for the 1st to
the last element.

It is possible to invert the heap order property so that P > X. In this case items in the heap
would be removed from largest to smallest. When using Heapsort the final sorted array
would be the opposite way around
Questions

1. What is a heap?
2. For a heap represented as an array. For any element at position X.

 What is the position of the parent of X?

 What is the position of the left son of X?

 What is the position of the right son ox X?

3. What is the heap order property:

4. Insert the following values into a heap: 44, 67, 88, 1, 5, 65, 43, 67

5. Delete the minimum element & re-order the heap

6. Using Heapsort sort the following values into ascending order: 5, 7, 12, 45, 66,
90, 2, 19

Suggested answers to these questions can be found at the end of this chapter.

Answers

1. A heap is: a binary tree that is completely filled except possibly for the lowest
layer, which is filled from left to right
2. For a heap represented as an array. For any element at position X.

 What is the position of the parent of X? X div 2

 What is the position of the left son of X? 2 * X

 What is the position of the right son ox X? 2 * X + 1

3. The heap order property is Key of parent X <= Key of X (except for the root)

4. Insert the following values into a heap: 44, 67, 88, 1, 5, 65, 43, 67, 4. The final
heap will be:
5. Delete the minimum element & re-order the heap
6. Again delete the minimum element & re-order the heap

Using Heapsort sort the following values into ascending order: 5, 27, 1, 25, 16,
90, 3, 19
The items are now deleted form the heap in the order, 1, 3, 5, 16, 19, 25, 27, 90.
The next three reduced heaps are given:
GRAPH

Introduction

There are many real world problems that can be modelled using a graph. Several are
described.

We will start by giving a standard set of definitions for graphs. The data structures most
commonly used will then be described and compared.

We are going to study several standard graph algorithms which can be used to solve a
range of problems. Shortest path problems for both un-weighted and weighted trees will
be discussed. Algorithms for graphs containing cycles will be treated, as well as ones that
do not contain cycles. Specifically Dijkstra’s algorithm will be discussed in detail. The
topological sort for acyclic graphs will be described.

The concept of a minimum spanning tree will be introduced and the solutions of Prim and
Kruskal will be described and illustrated. Finally activity graphs will be illustrated.
Minimum Spanning Trees

A Minimum Spanning Tree (MST) of a graph is a tree formed from graph edges that
connect all the vertices of the graph at lowest total cost. Most commonly one is interested
in the MST of undirected graphs. There are problems where one wishes to find the MST
of a directed graph but this is a more difficult problem & will not be covered here.

A simple example of a MST is the following: Consider a house with electrical points.
Each electrical point is a node of the graph. Each arc is the distance between the two
nodes. How can we wire the house with a minimum of wire?

Kruskal's Algorithm

Kruskal’s algorithm is a greedy algorithm that works as follows:

1. Order the edges from smallest to largest weight.

2. Accept an edge if it does not cause a cycle. Otherwise reject it.

Example: The original graph is:

AF 2 Accept
DE 5 Accept
EF 7 Accept
DF 8 Reject Cycle DEF
AE 11 Reject Cycle AFE
CD 13 Accept
EC 15 Reject Cycle DEC
CB 16 Accept

The Minimum Spanning tree is:

Kruskal’s algorithm is easy to implement manually. A computer algorithm is more
complex because of the problem of recognising when a cycle has occurred. An algorithm
to do this does exist. It is not described here because it is fairly advanced.

Acyclic Graphs

There are many graphs that contain no cycles and they are known as acyclic. Typical
examples are: prerequisite courses of some course and activity graphs. The latter are
essentially used to calculate the critical path through a network. Such planning is usually
used in any medium or large scale project especially in the construction business.
To find the minimum path for such graphs Dijkstra’s algorithm can be used. A simpler
way of doing this is by a Topological Sort which will now be described.

Topological Sort

1. Find any vertex with no incoming edges. Print it. Remove it & its edges from the
graph.
2. Repeat step 1 until there are no edges left.
For the above graph the final ordering is: CS1, M1, CS2, M2, CS3 (CS2 was arbitrarily
removed before MTH2)

This ordering is not unique. Basically different orderings will occur when it is possible to
validly remove more than one node at any point in time. The other 3 possible orderings
are:

 CS1, M1, M2, CS2, CS3

 M1, CS1, CS2, M2, CS3

 M1, CS1, M2, CS2, CS3

Questions

1. What is a minimum spanning tree?

Answers to this question can be found at the end of the chapter.

2. Use a Topological sort to get the paths (orderings) through the following graph.
For this sort what are the constraints on the graph?

Answers to this question can be found at the end of the chapter.

3. Use Kruskal’s algorithm to find the minimum spanning tree of the following
graph:
 Answers to this question can be found at the end of the chapter.

Answers

Answer to Question 4

A Minimum Spanning Tree (MST) of a graph is a tree formed from graph edges that
connects all the vertices of the graph at lowest total cost.
Answer to Question 5

There are 4 possible orderings.

 ADBCEF
 ADBECF

 DABCEF

 DABECF

Answer to Question 6
A-B 1 Accept
H-E 2 Accept
A-G 2 Accept
H-G 3 Accept
B-H 4 Reject Cycle A-B-H-G-A
E-F 5 Accept
F-G 6 Reject Cycle G-H-E-F-G
D-E 8 Accept
C-D 9 Accept
C-B 13 Reject Cycle C-B-H-E-D-C

Trees

Within computing the tree ADT is of fundamental importance. Trees are used when we
desire more efficiency in our data access or manipulation. We have seen that in the case
of a linked list we can achieve O(N) for all the relevant operations. This is very good;
however, if our tree is properly constructed operations such as insertion and deletion can
be implemented in O(log N) for N data items. This is a vast improvement and makes
sorting and searching algorithms, for example, more useful. The next sections examine
trees in great detail and highlight their benefits and deficiencies.

What is a Tree?

A tree is a collection of nodes each of which has a number of links/references to other

nodes. Each node will be the child of some other node, and we can thus talk about parent-
child relationships and so on. Figure 6 shows a tree, and it is clear from this
representation how this data structure obtained its name—it has a branching structure,
rather than the simple linear structure of a list. It is this branching that allows us to
achieve the efficiency we seek. But it comes at a prices: insertion and deletion into this
structure are more complex than for lists.
There is one node, the root, which does not have a parent. We can trace a path to any
node in the tree by starting at the root and following the appropriate links. There are a
number of other tree terms which we need to familiarise ourselves with:

sub-tree A tree is a recursive structure—a tree consist of a root node with a number of
other trees hanging off this node. We call these sub-trees. Each of these sub-trees can
again be viewed as a root node to which still smaller sub-trees are attached.

depth The depth of a node is the number of links we have to traverse from the root to
reach that node.

height The height of a node is the length of the longest path from the node to any leaf
node within the sub-tree.

leaf node A leaf node is one which has no children: its child links are all null references.

sibling If a node has a parent (the node that points to it), we call any other node also
pointed to by the parent a sibling.

The generalised tree that we have introduced here, in which a node may have an arbitrary
number of children, is rather complex to manipulate and implement. For most
applications a binary tree will suffice. A binary tree is a tree in which every node has at
most two children. That is, a node may have 0 1 or 2 children. The two children are
usually referred to as left and right. We also talk about the left sub-tree, for example, by
which we mean the sub-tree rooted at the left child node. It is important to distinguish
between the left and the right children since the order in which they are accessed will be
important for the algorithms we present below.

The remainder of this section deals with different kinds of binary trees.
Binary Search Trees

Trees are often used to obtain sub-linear time complexity for operations such as sorting
and searching of data. Database applications in particular make heavy use of complex
tree structures to speed up data queries. In order for data to be efficiently retrieved from a
tree structure we require an ordering property which is consistently applied when
building the tree. The simplest of these pertains to binary trees:

For any node X the data values in the left subtree are smaller than value stored within X ,
while the data values in the right subtree are larger than this value.

This is illustrated in Figure 7. The root node contain the data value 20. All the children
down its right subtree have values larger than 20, while those down the left subtree have
values less than 20. Since a tree is recursive (composed of yet smaller trees, in this case),
and this property is general, we see that the data respects this ordering property
throughout the entire tree! For example, we see that for the node containing the data
value 12, all its left children (only one in this case) are less than 12, while its right
children are greater than 12. The same property holds for the node with element 17. This
ordering property means that we can derive a very simple algorithm to effectively find a
data item within the tree. We usually conduct a search based on a key data value. Each
node will usually store such a search key value as well as some additional data (which is
what we wish to retrieve). In the examples we give here these values will be one and the
same.

BST Insertion

Insertion into a BST must maintain the BST ordering property. The algorithm is
surprisingly simple. Given a node with data value p that we wish to insert:
 1. If the tree is empty, link the root the the given node and exit.
2. If p is less than the current node data, proceed to its left child

3. otherwise, move down to its right child.

4. If the child node does not exist, slot the new node into that position and exit.

5. otherwise return to Step 2.

In other words, we branch left or right as we descend the tree looking for an empty slot
within which we can place our new node. The first empty slot we encounter will then be
filled; this involves linking the new node to the parent with the empty slot. This is
illustrated by Figure 8. We first insert 8, which simply involves linking the new node to
the root. Then, we insert the value 1: we see that 1 is less than 8, hence we move down
from the root node to its left child. This happens to be an empty slot, so we make the left
link of the root node point towards the new node (which is what we mean by “slotting in
it”). We then insert 13. Once more, we start at the root, compare the value to be inserted,
and drop left or right depending on this choice. Since 13 is greater than 8, we drop down
to the right child — which happens to be an empty slot again. We thus set the root node’s
right link to point to the new node and we’re done. To insert 12 we need to move further
down the tree. Starting at the root, we see that 12 is greater than 8, so we drop to its right
child. On the next step, we note that 12 is less than 13 so we drop to its left child. Once
again we have found an empty slot, so we set the left link of node 13 to point to our new
node. Finally, we insert the value 20. Starting at the root we drop to the right child (node
13) and then drop to the right again, landing in an empty slot. We then set the right link
of node 13 to point to the new node and we are done.
It is worth emphasising that the link that we use (left or right) when linking a new node to
its parent is not arbitrary: it is the link that we dropped down to get to the empty slot. In
Java programming terms an empty slot is represented using a null reference. After the
insertion, the reference is updated to point to the newly inserted node.

Implementation

In general we use recursive functions to implement tree-based routines. This often

provides a more natural way of looking at trees, since they are recursive structures. In the
case of insertion, this involves finding a base case (to terminate the recursion) and
deciding how we should partition the problem so that we can recursively solve a set of
smaller sub-problems.

To accomplish insertion we can write the following Java-like code, in which we assume
that we have a Node class with a left and right field which contain object references, and
in int to hold our data:

public Node insert(int x, Node parent)

{
if (parent == null)
return new Node (x);

if (x < parent.left.data)
 parent.left = insert(x, parent.left);
else
parent.right = insert(x, parent.right);

return parent;
}

This would be called with a statement like:

root = insert(3, root);

to insert the data value 3 into the tree. Note how a reference to the parent is returned at
every step of the recursion. We do this since the Node link (a Java reference) that we
went down in order to insert the Node will change when we hit an empty slot (a null
reference). By returning the value we pass back the new, updated, value to the function
that called this one. In the cases where no insertion happens the reference will be
unchanged so the assignment statements will not change anything.

We can see from the recursive implementation that

1. We have a base case—the first statement checks to see if the parent is a null
reference (we have arrived at an empty slot. At this point the recursion is
terminated and the function calls start returning.

2. We have recursive function calls (we call the function from inside itself) which
are shrinking the problem at each step. This is required for recursion to be useful.
Here we decide, based on the comparison, whether to insert the data into the left
or right sub-tree. We then call insert again with the new child node as the root of
the tree we wish to insert in. At the next function call, this check and decision are
made again, and this procedure repeats until we come to the base case (null
reference). Then we can start returning the data we need to get our final result
answer back

We see that when we insert the first item, a 3 in this case, with a root value of null, then
we return without any recursion at all. The root (which is a Java object reference will
now point towards a Node with the value 3. If we then proceeded to insert the value 4, the
following would happen:

1. root = insert(4, root)

2. root = insert(4,root){ root.right = insert(4,root.right)}
3. root = insert(4,root){ root.right= insert(4,root.right){
Base case: return NodeRef}}
4. root = insert(4,root){ root.right = NodeRef}
5. root = root
This is meant to illustrate the flow of logic in the recursive function calls: the braces
represent what happens within each function call. We can only return from the recursion
when we hit the base case. At that point we have enough information to return something
useful to the function that called us. For every function call we return the value of the
parent reference (which may have been changed). In this example the only reference that
is actually changed is root.right. The root of the tree is only affected by the first Node we
insert into the tree. If we inserted additional Nodes we could follow the logic in a similar
fashion, but it rapidly becomes cumbersome. The basic point to note is that we can
directly translate a recursive function into a Java recursive function implementation. As
long as we have an appropriate base case and provided we set up the recursive calls
correctly, the function will execute as expected.

BST Deletion

Deletion from a BST is somewhat more complicated than insertion. We must ensure that
the deletion preserves the BST ordering property. Unfortunately simply removing nodes
from the tree will cause it to fragment into smaller trees, unless the node is a leaf node
(has no children). The basic strategy is as follows:

1. Identify the node to be removed

2. If it is a leaf node, remove it and update its parent link;

3. If it is a node with one child, re-attach the parent link to the target’s node child;

4. If it is a node with two children, replace the node value with the smallest value in
the right sub-tree and then proceed down the tree to delete that node instead.

These cases are illustrated in Figure 9. The third case is the most interesting and bears
some discussion. The smallest value in the right sub-tree of a node is smaller than all
other items in that sub-tree, by definition. By copying this value into the node we wish to
delete, we preserve the BST ordering property for the tree. Furthermore, we know that
the node we now have to delete (the node we searched for down the sub-tree) can have at
most 1 child! If it had two children that would imply there was a node that contained an
even smaller value (down its left child). We thus end up back at one of the first 2 cases.
Implementation

The following routine shows how one might implement deletion. It assumes the existence
of a function called public int findMin(Node X) which returns the smallest data item
starting from the tree rooted at X, which is found by following the left link until you hit a
node with a null left child.

public void remove(int x, Node root)

{
if (root == null)
System.out.println("Data item is not in tree!");
else if (x < root.data)
root.left = remove(x, root.left);
 else if (x > root.data)
root.right = remove(x,root.right);
else // we're here! delete the node
{
if (root.left == null && root.right == null) // leaf node
return null; // this unlinks node by passing null to parent.
else if (root.left == null || root.right == null) {
// one child; link parent to node's child
return (root.left == null ? root.right : root.left);
else {
// two children - find min value, copy it over then delete that node
{
root.data = findMin(root.right);
root.right = remove(root.data, root.right);
}
}
return root;
}

Once more this function is a direct translation of the algorithm we described: we branch
left or right as we move down the tree looking for our target. Once we find it, we check
whether it has 0, 1 or 2 children and apply the appropriate deletion procedure. The return
statement in the single child case contains a compact way of generating a value based on
a simple “if-else” test. If the data item is not contained in a Node we will end up down
going down a null link; in that case we simply report this fact and we are done.

A proper Java implementation will need additional elements: a class called Node which
will contain the key/data and a class called BST which will define the structure of the tree
(a collection of linked Nodes). Here we have assumed we are inserting integers; a general
Java implementation would insert values of type Object. In that case the < operator can
no longer be used and a Java comparison method would be required to compare two data
values. We also need to store the root object reference within the BST class.

Tree Walks

While insertion and deletion are required to build and maintain the tree, the manner in
which binary search trees store data means that we can traverse them in interesting ways.
We talk about “walking” the tree, which simply means that we wish to visit each node in
the tree starting from the root. Generally we perform some or other useful operation as
we walk the tree, such as printing out the data value of the current node. We shall call this
the action in the following paragraphs. The action can, of course, be “do nothing”,
although would be rather silly!

There are 3 basic ways traversing the tree. They are all recursive, meaning that when we
refer to the root node, we can actually substitute any valid node within the tree.
InorderWalk: Starting at the root, we walk the left sub-tree. We then perform the action
on the root node. We then walk the right-subtree. Here, “walk the left/right sub-tree”
means that we apply the same procedure to the tree rooted at the left/right child of the
node we are on. Hence, the Inorder walk is recursive.

PreorderWalk: Starting at the root, we apply the action immediately. We then walk the
left sub-tree. Finally, we walk the right sub-tree.

PostorderWalk: Starting at the root, we walk the left sub-tree, then we walk the right sub-
tree. Finally, we perform the action on the node.

Level OrderWalk: A level order walk proceeds down the tree, processing nodes at the
same depth, from left to right i.e. level-by-level processing. This is also called a breadth-
first traversal. One can use a queue ADT to implement such a traversal, but we shall not
concern ourselves further with this scheme.

This all sounds rather bizarre, so an example is in order. The fundamental point to
remember is that the “walking” procedure is recursive: we apply the rules listed above for
each node we arrive at, not only the root node. The walking procedure is O(N) for N
nodes.

Consider Figure 10 and suppose we perform an inorder walk. The following steps occur:

1. We start at the root (node M).

2. We “walk the left-subtree” i.e. we drop to the node G.

3. The node G has its own left sub-tree: we “walk” that sub-tree dropping to A

4. We see that A does not have a left-subtree (it is a leaf node), so the “walk left sub-
tree” does nothing. Now, however, we can perform the action, since we have
processed the (empty) left sub-tree in its entirety. In this case, our action is simply
to print out the value of the node, “A”. We must now walk A’s right sub-tree. But
again, this is empty. So we have processed node A completely and can return
from that node.
 5. We have now walked G’s left sub-tree (visited every node), so we move onto the
action for that node: we print “G”.

6. We must now walk G’s right sub-tree - we move down to the node I.

7. As before, we have to walk the left sub-tree first: it is empty, so we are done.

8. Then we can perform the action: we print “I”. We then walk the right (empty)
sub-tree and we’re done with node I.

9. node G has now been fully processed.

10. We have now walked M’s left sub-tree (visiting all the nodes contained therein);
we perform the action: we print “M”.

11. We now walk M’s right sub-tree: we drop to node P.

12. We walk P’s left-subtree: we drop to N.

13. We walk N’s left sub-tree (which is empty); we can now perform the action for
node N: we print “N”; and walk the (empty) right sub-tree, which means we’re
done with node N.

14. We have now processed P’s left sub-tree; sow we print “P”; and move down to its
right sub-tree, starting at node Z.

15. Since Z is a leaf node, it has an empty left sub-tree; we thus print “Z”; see it has
an empty right sub-tree too and so we are done with node Z.

16. We are also at this point, done with the entire tree since we have processed each
node in turn.

We note when we look at the output that the inorder traversal has a return an ordered
(alphabetical in this case) display of the data in the tree. This is very useful. For example,
if the data in each node was a record in a database, we could use an inorder walk to
display an alphabetical list of all the clients’ information.

The pre and postorder traversals work in a similar fashion, but the sub-trees are walked in
a different order, so we will generate different output. In the pre-order case, we print out
a node’s data as soon as we reach that node, before moving on to recursively descend its
left and then right sub-tree. The printout we get in this case is

M, G, A, I, P, N, Z

which does not seem particularly useful. In the postorder case, we first process the left
and then the right sub-tree (recursively) before printing out the node value:
A, I, G, N, Z, P, M

Again, this does not seem to be of any use. Although it is not clear at all, there are many
uses for these tree traversals. Compilers in particular make heavy use of pre or postorder
walks to handle the correct evaluation of mathematical expressions in a programming
language. Directory tools for operating systems also make use of such traversals to
accumulate disk usage information and so on.

Implementation

The implementation for the traversal schemes is simple, if you view them recursively. As
usual, we require a base case to terminate the recursion, and we must ensure that each
recursive call works on a smaller portion of the input set (the tree nodes in this case). For
the the Inorder walk we have the following:

public void inorder(Node root)

{
if (root == null)
return; // Base case, walk an empty subtree, so do nothing

inorder(root.left); // walk the left sub-tree

System.out.println("node.data); // performed action (print, in this example)

inorder(root.right); // walk the right sub-tree

}

As we can see from the code, we can only perform the action on a node once we have
processed its entire left sub-tree. Having processed both the left and right sub-trees of a
node, we back up to the parent node and carry on working there. We thus move up and
down the tree as required by the recursive function calls. We never process “perform an
action” on a node twice.

The code for the preorder walk is as follows:

public void preorder(Node root)

{
if (root = null)
return; // Base case, walk an empty subtree, so do nothing

System.out.println(node.data);
preorder(root.left); // walk the left sub-tree
preorder(root.right); // walk the right sub-tree
}
Finally, for the postorder walk we have:

public void postorder(Node root)

{
if (root = null)
return; // Base case, walk an empty subtree, so do nothing

postorder(root.left); // walk the left sub-tree

postorder(root.right); // walk the right sub-tree
System.out.println(node.data);
}

While it is confusing and time consuming to follow the logic involved in these recursive
function calls, we note that, as before, there is a direct mapping from a mathematical/
intuitive definition onto a programming language. This is one of the benefits of recursion:
it allows us to express powerful algorithms very compactly and in an intuitive fashion.
Thinking recursively comes with time and practice, so do not be alarmed if it all seems
incomprehensible right now!

AVL Trees

While the BST is a good attempt at achieving sub-linear data manipulation, it has a major
flaw: it can degenerate into a list! In other words, depending on the data we insert or the
the deletions we perform our tree can end up skewed completely to one side. When this
happens the complexity of tree operations such as insert, delete and search rises from
O(log N) to O(N) and we may as well use a simpler list ADT.

Figure 11 illustrates the problem by inserting an ordered collection of values in to a BST.

Note that all the left sub-trees are empty.
Unfortunately we have no guarantee that data we wish to store will be randomized. In
fact, it is fairly likely that there will be some sort of ordering on the input data. Rather
than trying to concoct elaborate schemes to stabilize the tree, we adopt the approach of
imposing a balance constraint on the tree. In other words, we insist that, along with the
BST ordering property, the tree have some scheme to ensure that its height remains O(log
N). Ideally, we would want the tree to be perfectly balanced i.e. each left and right
subtree would have the same height. This is far too restrictive since we insert items one at
a time and a tree with even 2 items will, by definition, be unbalanced under this scheme!

A more relaxed scheme which nonetheless guarantees O(log N) height, was proposed by
Adelson-Velskii and Landis and is called the AVL tree.

In the case of an AVL tree, the height of the left and right sub-trees of any node can differ
by at most one. In this case the tree is considered balanced. One can prove that this
results in a tree which, although deeper than a completely balanced BST, still has a
height/depth of O(log N). This is all we require to make sure that the tree is useful for
data manipulation. An example of an AVL tree is shown in Figure 12; note the heights of
each node (see the earlier definition for tree height), and how it compares to the
corresponding “optimal” binary tree. It is worthing noting that there are many different
configurations, depending on the order in which the data is inserted into the AVL tree.

Insertion

An AVL tree uses a number of special tree transformations called rotations to ensure that
the correct balance is maintained after an insertion or deletion. These rotations are
guaranteed to respect the ordering properties of the BST. A very basic example of a
rotation is given in Figure 13.
After inserting A and B the tree is still AVL: the left and right sub-trees of the root node
differ by 1. Now we insert C. In this case we break the balance property at the root node
(the node B is still balanced): the left and right sub-trees of the root node now differ by
two. In order to restore balance at the root we “rotate” B towards A, resulting in the final
arrangement of nodes. We note two things:

1. the BST ordering property is preserved, and

2. the height of the root is now as it was prior to the insertion that caused the
imbalance.

This transformation is known as a single (left) rotation about B.We can also say that we
performed a single rotation with the right child of A or even that we rotated B towards A.

What happens in more complicated scenarios? As it happens there are only 4 possible
transformation we need to consider, regardless of the tree’s complexity. Two of these
transformations are symmetric counterparts (mirror images) so in reality there are only
two basic transformations: single and double rotations. We determine which one to use
by examining the path we followed during the insertion step.

Let us look at the single rotation first, Figure 14. As always, we assume that the tree was
balanced prior to the insertion. The figure shows one of the 4 possible scenarios in which
an insertion has violated the balance at the root node N1. Here, we have inserted into an
“outer” sub-tree, T(3) and this insertion caused the height of that this sub-tree to grow by
1 resulting in an imbalance at N1. To help us resolve the situation we expose another
node, N2: this is the first node on the path to our insertion point in T1. To fix the
imbalance, we rotate N2 towards N1 which involves rearranging the nodes and their sub-
trees as indicated. This single rotation is the generalisation of the example we introduced
above: the sub-trees T1 and T2 are empty there, while the sub-tree T3 consists of a single
node. Observe that the new sub-tree, rooted at N2 has the height it possessed prior to the
insertion. This means that we do not need to worry about the balance of the tree as a
whole: it has been restored to the (balanced) state it occupied prior to the insertion. We
thus see that this local tree operation ensures that the entire tree remains balanced!

This transformation has a symmetric counterpart (right rotation) in which the we simply
flip the figures from left to right. The transformation is shown in Figure 15.

What would happen if we inserted into the sub-tree T2 which then grew in depth by 1 and
violated the balance at the root? In this case, we see that a single rotation simply leaves
the height in its illegal state — Figure 16. We need an another transformation — the
double rotation. In this case we need to expose more of the tree’s structure. Once more
we choose the first few nodes on the path to the insertion point: we expose 3 nodes,
which we label N1, N2 and N3.
Notice that a single rotation can be used in the cases where the insertion has occurred
down the outside of the tree. A double rotation is only required when we have inserted
into an internal sub-tree. The double rotation rearranges the nodes and sub-trees in the
manner indicated—Figure 17—and also ensures that the balance of the sub-tree (and thus
the whole tree) is returned to its state prior to insertion. A double rotation can be viewed
as two single rotations: one from N3 towards N2 followed by one from N2 towards N1.

Essentially we use the following approach when inserting nodes into the tree:

1. Perform the insertion, noting the path we took to reach the insertion point;
2. Move up the tree from the insertion point and check for an imbalance;

3. If we reach the root and all is well, we’re done;

4. Otherwise, we check to see whether the insertion was into an “inner” or “outer”

5. Sub-tree of the left or right child of the unbalanced node.

6. For outer sub-tree insertion, we use a single rotation, otherwise we use a double
rotation.
Consider Figure 18. Note that the order in which we insert the nodes determines the
structure of the resulting AVL tree. Insertion of 1 and 7 do not cause any problems.
However, when we insert 12, the root node becomes unbalanced (indicated by a box). We
see that we have inserted into an “outer” sub-tree on the right child: we therefore require
a single rotation from 7 towards 1 in order to fix the imbalance. Note that subtrees are
indicated with triangles—empty sub-trees are shown when necessary so you can see
which rules are are using. After the rotation we have a balanced sub-tree rooted at node 7.

We then insert 10 and 11. Insertion of 11 causes an imbalance at node 12. We see that we
inserted into the right sub-tree of a left child node. This “zig-zag” pattern is characteristic
of a double rotation, so we resolve the right sub-tree further. In this case we have the
nodes 10, 11 and 12 which will be involved in the double rotation. All the sub-trees are
empty, however. This does not matter! We rotate from 11 towards 10, and then from 11
towards 12, giving us the tree shown on the right. As before, this transformation restores
the local balance of the tree to its state prior to insertion, so we do not need to fix things
further up in the tree.
We now insert 15. This causes an imbalance at the root. We see that we inserted into the
right sub-tree of the right child of the root. Such an “outer” insertion is characteristic of a
single rotation. We rotate 11 towards 7 to fix the tree. We then insert 8 and 10.5 (the fact
that’s its a real number is not important). The tree remains balanced until we insert 9.
This introduces an imbalance at the node 7. As before, we see that the insertion took
place into a left sub-tree of the right child of 7 — a zig-zag pattern. This tells us that we
need to use a double rotation to fix the imbalance. The nodes involved will be 7,8,10,
with the sub-trees as indicated in Figure 19. The resulting tree is shown on the right-hand
side of the diagram.

Implementation

An AVL Node looks much the same as BST Node, but each node now maintains a height
field, which is the height of that node. By looking at the heights of left and right children
Nodes one can see whether a Node is unbalanced and restore the tree. Remember that we
only require a single or double rotation to restore the tree completely. The basic
algorithm is as follows:

1. Perform recursive BST tree traversal to find the insertion point (first null Node)
2. Insert the New node
 3. Recalculate the height of tree into which we inserted

4. As we back out recursively, check the heights of each left and right child node

5. If they differ by more than 1, apply the appropriate rotation, and we are done.

Note that once we have fixed the unbalanced sub-tree we are done with our task, but we
still have to return from all the recursive function calls we used during the insertion.
However, we know that the balance at all other nodes will be OK, so we will never have
to do another rotation. It is not considered good programming practice to prematurely
terminate the return of a sequence of recursive calls: if efficiency is critical, you can use
an iterative (non-recursive) implementation, but this will require more complex code.

Deletion

Deletion is significantly more difficult, if you apply the formal algorithm. However, all
you really need do is the following (at least when you’re working out examples by
hand!):

1. apply the Standard BST deletion algorithm to find and delete the node,
2. starting at the deletion point, move towards the root, checking the node balance at
each step,

3. if you find a node out of balance, apply a single or double rotation to fix it; to
determine which to apply,

a. identify the sub-trees attached to the left and right children of the
unbalanced node;

b. note which sub-tree is deeper, an internal or external one (in the former
case, resolve more structure);

c. identify the Nodes Ni leading to this sub-tree and then apply the fix;
continue upwards.

4. you may have to apply a rotation at each node as you continue towards the root.
The root itself may have to be rebalanced.

As this algorithm implies, in the worst case you would incur O(log N) rotations to
rebalance the tree after a deletion. This is not ideal, since a rotation requires a fair amount
of logic to implement.

Consider Figure 20. Remember that we apply the standard BST deletion strategy to
remove nodes from an AVL tree. We begin by deleting 10 (which is a leaf so this is
trivial).This causes an imbalance at node 20. To identify the kind of rotation we require
to rebalance the tree, we see that the left-subtree of the right child of 20 is deeper than the
right sub-tree (which is empty). We thus require a double rotation, resulting in a new sub-
tree with a root of 22. Unfortunately, as we continue moving up the tree we see that out
manipulations have unbalanced a node higher up! In this case the root of the entire tree is
the culprit: we can at least be sure that no additional rotations will be required once we
restore the balance of the root. We identify the left and right children of the root node,
and look at the structure of their sub-trees. We see that the outer-most sub-tree is the
deepest, which immediately tells us that a single rotation will suffice. We thus rotate from
80 towards 30, fixing the balance of the root node and thus terminating the deletion
procedure.

We then wish to delete 30. We see that 30 has 2 children: we thus invoke the BST
deletion strategy and replace 30 with the smallest value down its right sub-tree, before
proceeding down the sub-tree to delete that node. In this case the tree balance has not
been affected. We now remove 70. This is a leaf, so we can delete it easily enough, but in
doing so we unbalance the node 60. Once more, we identify the sub-trees of the left and
right children of this node and see that we can get away with a single rotation from 22
towards 60. Note that this rotation does NOT reduce the height of the sub-tree, but it is
sufficient to ensure that the AVL property is restored.

We then delete 25 (a leaf) which does not damage the tree. Deletion of 80 requires that
we replace it with the smallest value down its right sub-tree and proceed to delete that
node. We thus copy 90 into 80 and delete the node 90 in the right sub-tree. As usual, we
use the BST deletion process: we simply link the original 90’s child to its parent,
“bypassing” the node we wish to delete. Deletion of 22 follows the same logic.

Finally, we delete 95, 110 and 100 in succession. The final deletion causes the tree root to
become unbalanced. We see that an outer sub-tree is deeper and thus perform a single
rotation from 60 towards 90.

Implementation

The formal implementation of deletion involves a fairly large number of test cases
according to which you select either a single or double rotation. Unfortunately, unlike
insertion, fixing a local sub-tree after deletion may not return its height to what it was
prior to that deletion. This means that the tree may become unbalanced further up,
requiring still more rotations. The standard implementation is recursive (as one would
expect), and uses the usual branching tests as you move towards the deletion point. It
mirrors the BST deletion code, until you actually perform the deletion. In this case the
various cases for fixes via rotation have to be enumerated, and the fixes applied. We are
guaranteed that after we do this the sub-tree rooted at the node out of balance will be
balanced once more. We can then recompute its height for later use and recurse back up
the tree. One tricky aspect of the code is the way in which you recalculate the node
heights efficiently as you recurse back up the tree.

There are numerous Java code implementations performing both insertion and deletion
and rather than getting bogged down in details, we refer the reader to those. In practise,
the more efficient Red-Black tree ADT is used to implement a balancing scheme.
Unfortunately this is a rather sophisticated data structure, which falls beyond the scope of
these notes.

Other Tree Representations

There are a multitude of tree ADT’s used for data storage and manipulation. The reason
is obvious: a well balanced tree reduces access times from O(N) (for a sequential scan) to
O(log N). To achieve this we need to constrain the insertion and deletion procedure. We
have already seen one way of doing this — the AVL tree. Two other popular methods
are:

Red-Black Trees: A Red-Black tree uses a node colouring property to enforce a balance
constraint. Nodes are coloured either Red or Black as they are inserted, and the insertion
and deletion procedures are modified to ensure that a consistently coloured tree emerges
at each step. A Red-Black tree obeys the following rules:

1. The root node is coloured black;

2. We may not have two consecutive red nodes;

3. every path from the root to a null reference has the same number of black nodes.

A Red-Black tree tends to be a little deeper than an AVL tree (still O(log N)), but has the
benefit that deletion will not require transformations all the way back to the root. It is
now the preferred data structure for binary tree manipulations.

B-Trees: A B-tree is an M-ary tree which satisfies a set of constraints. An M-ary tree is
one that can have at most M children per node. A B-tree differs from most tree’s in that it
grows level by level. The tree properties are as follows:

1. data is stored in special leaf nodes;

2. non-leaf nodes contain at most M - 1 search keys, and M node references;

3. root is either a leaf node, or has 2 to M children;

4. other non-leaf nodes have between M / 2 and M children;

5. leaf nodes are at the same depth and contain L / 2 to L data entries.

The values M and L are chosen based on the application, but are usually related to disk-
block size. B-trees are the primary data structure used in large database systems since
they ensure very fast query times. For a B-tree we have O(logMN) for N data nodes, due
to M-way branching.

Lists

A list is a data structure in which data items are accessed by moving through the
collection from start to end. It is thus a linear data structure since the algorithms used to
traverse and manipulate the list have complexity O(N), for N data items. The list is an
example of an abstract data type (ADT): a data structure (or object in programming
terms) with an associated set of algorithms to manipulate it. The concept of a list does not
impose an particular implementation. However, from a programming point of view some
implementations are better than others, an these have become the norm. We will examine
two of these: singly linked and doubly linked lists.
Singly-Linked Lists

A linked lists is simply a collection of nodes (which will hold item data, at the very least)
which are connected in a special way. The use of a data-bearing node is common to all
the data structures or ADT’s that we will see; it is the way they are connected and
accessed that differentiates them. In the case of a singly linked lists (linked list for short)
we simply “link” each node to its successor. By successor we mean the node that
logically succeeds a given node: usually we impose an ordering on the data in the list, so
a “lower” value would precede a “higher” value. This is not required by the definition
however. As far as an implementation is concerned, we have have a node object, within
which we store a reference to its successor, as indicated in Figure 1. Note that a null
reference (one that is not pointing to anything) is represented by lambda in the figures.

The node will store the actual data item, and may hold other information too. The final
node in the list will contain a null reference since it has no successor. Finally, to complete
the representation, we need a means to identify the front of the list, which requires a
reference to the node from which we can reach all others. We can call this the head of the
list.

Insertion

Insertion can take place in a number of different ways: internally (within the list) or at
either end. The basic idea is simple enough:

1. create a new node with the given data item

2. find the successor and predecessor of this new node in the list

3. insert the node at this point by resetting the links appropriately

There are unfortunately a number of complications. When we insert at either end of the
list we are missing either a successor or a predecessor. The code that you write has to to
check for each of these 3 cases and must link the new node in the correct way. The
different possibilities are indicated in Figure 2. Finding the position to insert the new
node is O(N).
Delettion

Deletion involves identifying a node in the list which is to be removed (an operation that
is O(N))), and then unlinking this node from its successor. The predecessor of the node is
then linked to the node’s successor. As was the case with insertion, there are 3 possible
cases to consider: deletion from either end or deletion from within the list.
Doubly-linked Lists

In many cases it is necessary to move through a list of objects from either the front or the
back of the list. In this case a single-linked list becomes less attractive. If we wish to
move through the list from the back, we must first identify the end of the list (which
requires N steps from the head reference) and then somehow move towards the front of
the list. This involves keeping track of the predecessor to each node as we step back
towards the head. One can, of course, maintain a reference to the last element of the list,
which we can call the tail. But even in this case we need to maintain a reference to the
predecessor as we move backwards through the list.

A better approach, which leads to a cleaner more efficient implementation, is a

doublylinked list. In this case our node now contains two references in addition to the
data: a reference to the successor and one to the predecessor. When we insert or delete a
node we now have to update more references, but the net result is a list that can be easily
traversed in either direction and has constant time access to the head and tail elements.
This is shown in Figure 3.

Insertion

Insertion for a doubly-linked list is similar to the singly-linked case: we may insert at
either end or in the middle of the list. Insertion at either end is trivial we update the head
and tail references and make the new node’s successor or predecessor point to the
original first/last node. Insertion in the middle of the list requires us to identify either the
successor or predecessor of the new node within the list, and then to update the nodes
involved so that there predecessor/successor links are consistent. These operations are
shown in Figure 4.
Deletion

Deletion in a doubly-linked list is similar to the single-linked case: we identify the node
to remove, and then simply update the predecessor and successor references to reflect the
fact that the node has been removed. As before, we need to write code that will deal with
the 3 cases: deletion from either the head or tail of the list, or deletion from the middle.
Some of these operations are shown in Figure 4.
Analysis of Algorithm

An algorithm is a specification for a accomplishing a given task. The program that a

computer executes is an algorithm, albeit a rather long and complex one. In computer
science terms the word algorithm usually refers to a specification for a solution to an
important class of problems. For example, a means of arranging data items in a way that
allows efficient access. These are the sorts of algorithms we will concern ourselves with.
In this context, it becomes important to have model of the time and resource requirements
that a particular algorithm imposes on the system. That is:

1. how much memory will it require to accomplish its task, and

 2. what length of time will we need to wait before we see the results?

These issues are addressed through something known as algorithm analysis.

Algorithms are often naturally associated with a data structure. A good example of this is
the concept of a queue. As we know from experience, people arrive at a queue from the
back and leave from the front. Thus, the model — or abstraction — of a queue is of a
sequence of items which can only be accessed from the back or front. Given this
abstraction, we can design a data structure which represents this behaviour, and then
develop the necessary algorithms to access and manipulate such an entity. The data
structure needs to encapsulate the idea of queueing at the back and dequeuing from the
front. Because we are thinking in abstract terms, it does not matter what sort of items are
actually in the queue: we are simply deciding how it should be represented, and how to
manipulate it. Of course, if we need to create a real queue, we need to know what it will
contain. However, if we know how a queue will operate in this abstract sense, we
immediately know how it will operate for any type of item! Thus, once we design the
data structure itself we can develop algorithms to access and manipulate it, without any
regard for what it will eventually contain. The data structures we will consider are
commonly used throughout computer science, since they are abstractions of very useful
concepts related to the efficient storage and access of data.

Algorithm Analysis and Recursion

Before advancing to the description of data structures and their associated algorithms, we
need to spend some time familiarising ourselves with so-called complexity analysis.
Some data structures are inappropriate, from an efficiency point of view, for certain tasks.
For example, the queue referred to in the introduction would not be a good way of storing
items if we wish to search through through it to find a specific item. Algorithm analysis
provides the tools we need to estimate both the memory and time requirements (or
complexity) of the algorithms we wish to examine. As part of this process we will briefly
examine the concept of recursion and discuss the perils of using recursive algorithms
arbitrarily.

Notation

In general, the algorithms we are interested in will manipulate a number of data items,
arranged in some sort of data structure. Let us call this number N. For each of our
proposed algorithms, we can derive a function T(N) which will show how the complexity
(space or time) of that algorithm is tied to the number of data items. For example we see
that, given N items arranged in a list, we require an average of N / 2 steps down this list
to find a given item. We could thus say that the time complexity is

T(N) = N / 2

Of course, the actual computer program that performs this “list traversal” will require
many more instructions to move through the list and perform the various tests that are
required. However, this extra overhead will simply result in a larger value multiplying N
as well as an additional constant. Thus the expression remains a polynomial of order 1—
this is a linear relationship (of the form a * N + b).

When we analyse the complexity of an algorithm, we do not wish to consider issues such
as the precise value of the constants referred to above. Usually we wish to see how the
algorithm will perform for very large N, since most of our data structure will need to
accommodate large amounts of data. So far we have only seen a linear expression for
T(N). Unfortunately for most algorithms the complexity is somewhat worse: we may
have a quadratic function, for example. If we wish to compare two different algorithms
for accessing a data structure, for example, we need a way of comparing them sensibly.
Given the following two complexity estimates,

T1(N) = 1000N + 400; T2(N) = N2 + 3N

which one would be better? The answer: it depends! For small values of N, T2(N) will
win out, while for even moderate values of N, T1(N) will be better. Intuitively, however,
we would expect a linear expression to give use a lower complexity estimate that a
quadratic expression. In particular we note that, for large values of N:

1. constants become irrelevant;

2. only the highest order term remains significant.

What we want is a way of sensibly comparing two functions so that we can see which is
“best” for our needs. The system we use must provide the correct ordering of functions,
regardless of lower order terms and constants. We can achieve this by introducing the
following notations:

Big-Oh A function T(N) has complexity O(f(N)) (pronounced Big-Oh), written as

T(N) = O(f(N))

if there are positive constants c and n such that T(N) <= c * f(N) when N >= n.

Big-Omega A function T(N) has complexity Omega(f(N)) (pronounced Big-Omega),

written as

T(N) = Ω(f(N))

if there are positive constants c and n such that T(N) >= c * f(N) when N >= n.

Big-Theta A function T(N) has complexity Theta(f(N)) (pronounced Big-Theta), written

as

T(N) = θ(f(N))
if and only if T(N) = O(F(N)) and T(N) = Omega(N)

Little-OhT(N)= o(f(N)) if T(N) = O(f(N)) and T(N) != θ(f(N))

The first notation, Big-Oh, provides an upper bound when comparing functions: we are
guaranteed that T(N) will always have a lower (or equal) complexity when compared to
the function f(N). In many cases this is all we require, since we wish to estimate the
worst-case performance of our algorithm. If we can show that our algorithm is O(N) for
example, then we know that we can handle very large numbers of data items without
running into trouble. An algorithm with O(N3), however, is essentially useless unless we
can guarantee that N is very small.

The other notations are less frequently used, but provide other sorts of ordering
information. Big-Omega provides a lower bound estimate i.e. our complexity is guarantee
to either match or exceed f(N), while Big-Theta provides a precise growth rate for the
complexity: f(N) is bother an upper and lower bound. Finally, Little-Oh is simply Big-
Oh, but with the requirement that the growth of T(N) is strictly less than f(N).

Table 2.1. Relative ordering of functions. In practice algorithms with complexitites above
quadratic are deemed to be too expensive.

Function Name
C constant
log N logarithmic
log²N log-squared
N linear
N² quadratic
N³ cubic
2N exponential

The table above shows a list of functions ordered by growth rate, and is a useful one to
remember.

When you are comparing functions using any of the above definitions note the following:

 do not include constants or lower order terms i.e. T(N) = O(N²) rather than T(N) =
O(3N² + 2N);
 it is clear that for T(N) = N² both T(N) = O(N³) and T(N) = O(N²) are valid. You
should choose the tightest (most accurate) bound.
Rules for Code Analysis

There are a few formal rules that can be used to aid in the analysis of algorithms, as well
as some less formal more intuitive rules that make code analysis easy. The formal rules
are:

Rule 1: if T1(N) = O(f(N)) and T2(N) = O(g(N)), then

1. T1(N) + T2(N) = max(O(f(n)), O(g(N)));

2. T1(N) * T2(N) = O(f(N) * g(N))

Rule 2: if T(N) is a polynomial of degree k, T(N) = Theta(Nk).

Rule 3: logs are sub-linear: logkN = O(N).

These formal rules help us to arrive at the short-cuts indicated below. We tend to use
these short-cut rules when evaluating the Big-Oh complexity of a piece of code we have
written.

FOR loops: The number of operations completed within a FOR loop is simply N times
the number of operations at each step (N is the loop counter). Such a loop is thus O(N).

Nested FOR loops: A nested FOR loop executes the instructions in the innermost loop Nk
times, where k is the number of nested loops. It is thus O(Nk). Note that if you use
different variables for loops counters, the complexity becomes a little more difficult to
determine but is still the number of times instructions are executed within the innermost
loop.

Consecutive statements: The complexity of a sequence of statements is simply the sum of

the complexities of each individual statement;

Branches: for a branch instruction (if/else, say) we look at the two complexities for each
branch and take he largest of these.

An example will help to clarify things. Given the following code snippet, estimate its
Big-Oh complexity:

public static int whatsitdo(int N, int p)

{
int sum = 0, i, j;

sum += N;
sum = sum + 2;

if (sum > p)
for (int i = 0; i < N*N; i++)
 sum += i*i;
else
for (i = 0; i < N*N; i++)
for(j = 0; j < N; j++)
sum += i+j;
}

We see that we start with a sequence of statements: two lines of code, then a singly
nested loop followed by a doubly nested loop. The number of items is N, so we derive
our time complexity expression using this variable. According to the rules stated above, a
singly nested loop has complexity O(N); however, in this case the loop counter ranges up
to N², and we thus arrive at an upper bound of O(N²) for the first loop. The doubly nested
loop would have complexity O(N²), but now we have O((N²) * N) or O(N³). The first two
statements require a constant time independent of N , giving us O(1). Finally, the branch
instruction could go into either the single or double loops; we thus take the larger part
(the else branch). Adding all this together we get O(1) + max(O(N²), O(N³)) = O(N³).
This algorithm is cubic in N and would thus be a very poor bet for any reasonable size N!
We would then go back to our design and either try to design a better algorithm for our
data structure, or create a new data structure which might better support the operations
we wish to perform.

Recursion

Recursion is widely used in many of the more elegant algorithms required for efficient
manipulation of data items. The basic idea may seem strange at first but it the concept is
mathematically well defined, so even if it bothers you, it does rest on a sound basis!

Let us first consider recursive functions in a mathematical sense. If we have a function

T(N) we say that it is recursive if it can be expressed in terms of itself. That is

T(N) = F(T, N)

While this would seem to lead to a chicken-and-egg type scenario, this is not so, provided
we are very careful when defining the function F. The simplest example of recursion is
the factorial function:

T(N) = N*T(N - 1); T(0) = 1

We see here that a recursive function must have a so-called base case to allow the
recursion (self reference) to terminate. If there is no base case, the function will expand
into an infinite definition. The other point to note is how the self-reference is achieved:
we cannot use T(N) on the right hand side, since we have not calculated it yet. However,
we can use terms that have been calculated—in this example T(N - 1)!
To show how one would evaluate such an expression, let us trace the steps to evaluate
T(3):

1. T(3) = 3T(2); we now need T(2)

2. T(2) = 2T(1); we need T(1)

3. T(1) = 1 by definition, thus

4. T(3) = 3( 2( 1 )) = 6 and we're done!

The idea, then, is to build up your solution based on previous data you have calculated.
Many functions can be defined recursively. In particular, algorithms designed for data
structures which are self-similar (we’ll see this later) make heavy use of recursion. For
such recursive algorithms we can define recursive relationships to estimate the
complexity, but this is a fairly advanced topic that we will avoid.

The reason we are so interested in recursion stems from the fact that almost all computer
languages in use today support recursive function definitions. We can thus design simple
and elegant recursive functions to solve a whole host of problems, secure in the
knowledge that we can code them up directly. A recursive function definition for factorial
might look something like:

public int fact(int N)

{
if (N <= 1)
return 1;
else
return N*fact(N-1);
}

The system will arrange for the function fact to be called with different arguments as it
recurses down towards the base case. At that point, we have all the information we need
to start evaluating the partial expressions and as a result we can evaluate the original
request. Just as we did for the mathematical definition, we require a base case to halt the
recursion. If you do not have such a check, the system will go on calling the function
until it eventually runs out of memory.

While recursive functions are “cool” they are not always appropriate. When a computer
implements a recursive function call it needs to save extra information to help it resolve
the call correctly. The more recursive calls we need, the more resources it has to expend
during evaluation. For the case of the factorial function, we can re-write it using iteration
(i.e. a loop), which is computationally equivalent but less resource intensive:

public int fact(int N)

{
 int f = 1;
if (N <= 1)
return f;
else
for (int i = 2; i <=N; i++)
f = i*f;
}

In the case of the factorial function recursion is still a valid, if expensive, alterative.
However, for some functions a recursive definition imposes additional overhead because
data that has already been calculated ends up being recalculated. This is wasteful and
leads to a very slow recursive solution. This is illustrated by the following recursive
function (the Fibonacci Series)

T(N) = T(N - 1) + T(N - 2); F(0) = F(1) = 1;

In this case a naive recursive implementation on a computer causes the program to slow
to snails pace. Consider the function call T(4), from this we need to calculate T(3) and
T(2) , but to calculate T(3) we need to calculate T(2) and T(1). We thus end up
calculating T(2)(recursively!) a second time! If you compound this recalculation over
many different function calls it results in an enormous overhead. In this case, one can
again use an iterative approach to calculate the desired number T(N). In fact, the
complexity is O(N) versus the exponential complexity of the recursive case. The basic
lesson we learn from this example is that recursion is only worthwhile if we do not
duplicate work. We shall see a great deal more of recursion in subsequent sections.