Access Method I/O

CHAPTER ‘OBJECTIVES

When you complete this chapter, you will bc able to:

° Describe the user visible file and record operations as well as when and how

each is used '

` ~ Understand the concept of a current record and how it is used by both the ac-

cess method and the user`s program '

' Understand the internal control structures that are required to design and im~

plernent an access method '

° Discuss the critical aspects and trade-of fs in l/O perforrriartccand how overall

throughput can be improved dramatically without changing one line of code

in the user‘s program

° Discuss how accesslmerhod 1/O differs from basic file system I/O

8. 7 0 VERVIEW

ln this chapter we discuss how an access method (AM) spans the gulf bctwccn the

- user and the tile system in order to get the job done. To accomplish this objective,

we must determine what kinds of operations users would like to be able to perform

on records. Then we will examine how these requests can be mapped by thc access

method into a series of calls that are understandable by the file system.

188 Access Method UO Chap. 8

The access methods form the standard interface between applications written
in high-let el languages andthe file system/devices (see Figure S»0). As such, the AM

also serves to hide most, if not all device specifics from the application, and thus

the user really only needs to understand the application tp be done. ln addition, the

;,¢.;,~5, mgrhod is able to provide user programs with more capability than the basic

tile .system alone can provide. Thus, although many ofthe commands appear similar

to the BFS commands. the user is really gaining the capability to work with records

instead of blocks. The specifics of the access methods will become much clearer in

Part lll of the book. Finally, the AM understands the content of :he blocks that

iz passes to the file system lo be read or written. The file system, on the other hand,

does not understand the content of the blocks that it reads and writes.

8 2 FILE DESIGN CONSIDERATIONS

How does an access method actually organize user data records within a particular

file organization? What algorithms are used to insert new records and also to retrieve

old data records? Each ofthe tile organizations discussed in the rest of Part lll uggg
different techniques and trade~offs. lt is the purpose of this section within each chapter

to discuss in detail how each file organization is designed. , _

8.2. 1 File Attributes , > ,

ln this chapter we .discuss the file-organization-independent lilc attributes that are

required to perform a particular function call. Then, in each of the following access.

method-specit'tc chapters, we discuss only those attributes and characteristics that

are required over and above this norm.

For example, the standard set of tile attributes might consist of the following

IICHISI -

° Filename or pathname

' File organization type

~ Record format and maximum record size

~ Maximum block size

° File size

° File organization-specific' information

It is the last item inahe list above that will be discussed in some detail in each

of the following access method chapters.

8.2.2 File Structufe . s

To insert and retrieve user data records from a file, it is necessary to build enough

file overhead or Iile structure to enable the associated access method to access the

data records. This is similar to the overhead that the basic tile system built on the

disk or tape devices. On disk, the BFS built a boot block, a disk allocation table,

and a directory structure. With this system overhead, the BFS was able to manage

all of the users and all ofthe disk space efficiently and effectively.

This section in each of the unique access method chapters that follows will discuss

what structure is built and why. With knowledge of the file structure, we will im-H

be better able to understand the following characteristics of files: '

‘ The performance characteristics of each lilc organization

° The space requirements for each type of file

- The benefits as well as drawbacks for each file organization

8.3 FIL

190 Access Method I/O Chan- 3

E OPERA 7'/ONS

\\'h;ir kinds of operations do users want, to perform, and what kinds of file opera-

tion; med 10 be supported? File operations operate not on the data records within

the me but on the tile itself. For example, we might want to create a file today and

a¢¢¢55 of 0/,gn it tomorrow. In addition, when we are done processing the file, we

mav want to cut the link to the file or close the files Similarly, we may not always

zue-ss correctly how large the lile will become. Therefore, we will want the capability

of increasing _the size or extending the tile. Finally, at some point in the future, we

may no longer need to keep it; and therefore we will want to get rid of or delete the file.

ln the following sections we discuss the requirements and functionality of these

tile operations, As we proceed through this section, try to relate the functions and

information requirements to the capabilities provided in one of the higher~level

languages with which you are familiar. When you run across data that are not sup-

ported by the language, try to determine where the access method can retrieve the

information. Finally, compare these access method file operations against those per~

formed by the basic file system. _

8.3. 1 Create File

The purpose of the CREA T_Efunc1io/1 is ro allow users lo create brand-new files.

Before any records can be written into a file, and before any records can be read

from a file, the file must first be created. How can a file be created? What does it

mean to create a file? If we examine most high-level languages, the capability of creat-

ing all but the most simple sequential files isjust about nonexistent. One of the reasons

for this is that the languages were defined with a particular audience in mind.

» COBOL was designed for efficient business applications. These applications

tend to perform many I/O operations. Thus the COBOL 1/O support is fairly exten-
si\'e_ FORTRAN was targeted at scientific and engineering applications that require

many calculations and few l/O operations. Thus the FORTRAN language supports

,fr very restricted set of l/O operations. Given this situation, most systems provide

users with utility programs that interactively request all the data needed to create

a lilc_ and then it goes off and creates the file itself.

What information is needed to create a f1le'!. ln addition to the information

needed by thc BFS CREATE_FlLE function, all of the following data are needed:

° Device/user name or pathname

~ Filename

‘ File 'size

~ File organization

Block sizfi A

° Record format

' Maximum record size

File-sharing information

' Filecrganization-specific data

Next, we discuss step by step the tasks performed by the CREATE function.

I. Validarg All lnpulparamelenr. Within a data management system, we must

never create a tile that cannot be used because the user parameters are incorrect, or

worse, correct but inconsistent. At the successful completion of the CREATE re-

quest, the user must be assured that the lile is ready for use. Therefore, the job of

the access method is to ensure, to the best of its abilities, that the input parameters

are consistent' and correct. For example, if a filename was not supplied, how could

the user identify and locate the file tomorrow? Also, it is wrong if the block size

specified for the lile is smaller than the record size, because a block must be able

to contain an entire record. -

2. Call the Basic File Sysrenr ‘s CRE/I TE_FlLE Function. - The heart of

CREATE processing requires that the correct disk or tape be mounted in the device,

the name of the file be added to the directory, and disk space be alloested to it. These

are the primarygfunctions of the basic file system CREATE_FlLE routine. ln addi-

tion, the I/O buf |'er(s) must be allocated and initialized. The size of each l/O buffer

is 'determined from the block-size input parameter." f

3. Perform File-Organization-Specific Processing. ln virtually every tile

organization, there is something “nonstandard” ,that must be done in order to prop-

erly CREATE the tile. In this section we discuss those things that must be done for

each file organization over and above the standard processing, For examples, labels

could be written onto tape tiles. - `

4. 'Write Out the l~7le's Attributes. As part of the CREATE processing,

information must be written onto the medium in order for the file system to locate

the file and to read and write the blocks within the file. ln addition, information

must be written out in order for the access method to be able to manipulate the records
within the tile. The basic tile system deals only with the blocks in the file, but the

access method must deal with the records within the blocks.

For the access method to be able to manipulate the records, it must know the

record‘s format and size as-well as other file-organization-specific data. The file

organization, record format, and record size are attributes of the file that the access

method must have in order to operate on the records correctly. Therefore, this information
must also be written to the file. ln fact, the overall set of file attributes, as

these parameters are really known, include all the important information about a

tile that is needed to be able to process the file correctly, such as:

.` ~

° Filename

. .1

° File organization

,‘ Block size

‘ Record format
Record size

' file size `

° File organiiation specific Clara

The question is: Since it is soaimportant, where is this information written? The

jile sysle/ri can record these dam virtually anywhere, as long as they can be readily

retrieved whenever they are needed. The two easiest places to record the data are

giihgr as part of the lile's directory entry or within the file itself. The directory entry

is acceptable for standard data required by all lile organizations, but when we get

to me-specific; information, we begin to run into complications oi' size of entry, as

well as directory search performance degradation issues. Therefore, we will adopt

the convention of putting only a minimal amount of data into the directory entry

and the rest of the file attributes into the file itself (see Figure 8-1). For example,

in the directory entry we could store the following infomation about the file:
Filename

* File organization '

' Address of first block in the file

- Fate size `

° Block size

Then, within the Hle itself we could store the "nonstandard" data, such as:

~ File space allocation

e Maximum record size

° Record format '

° File-organizatiorvspecific data

' ' The rationale for t/tLs_spIit is to put into the directory entry only those data

that might be reasonably useful when not acc'e.rsirtg_the_/ile. After all, if we were go-

ing to access the me, it' would not really matter inuch where the data had been stored,

sitice we would have to retrieve them anyway. However, if we wanted to request a

ldirectory listing, it might be useful to have some of the data about the file displayed

without taking the performance penalty of accessing the file. ~ --

Where in the file do we write the attributes? By convention, we can record them

in the first t1 blocks of the file. Why? Because at create time, there are no records

in the file yet, and thus the first block in the file will always be available. Also, it

is an easy convention to remember and the file system dos not need to go looking

through the me to and me attributes.

‘ 5. Return Status and a-File Identtyier to Calling Program. This is the last

step in the file CREATE prooess. The file identifier is required so that the user can

read and write to many different files atfthe same lime. The file identifier allows the

‘ user to uniquely specify which of the current (iles the record operation is to be per-

formed on. lt is always necessary to notify the calling program whether or not the
request that was made completed successfully or whether it encountered errors. Only

when thecalling program is told that the operation completed successfully can the

program assume that records can now be written into the file.

Unsuccessful CREATE requests could have failed for any number of reasons,

such as:

¢ Invalid create input parameter (e.g., no filename specified).

' No space available to create the file on the medium selected. _

° File already exists on the medium with the same name. ln this case either we

gave an incorrect name, or the file had already been created by someone else,

What is important to remember is that the access method is the interface to

the user application programs, and status must always be returned to the user`s

program.
Iii \t|iiim;iri; we haw seen that the access method (.~\:\l) CREATE function

grcatl; o\crl:tp_» the basic file systenfs CREATE_FlLE operation. \\’e can summarize

this ;i\ follousi

_.\_\t CREATE = (BFS CREATEAFILE) + (AM-specific processing)

.\i mc gompletion ofthe CREATE processing. the file has been completely built

and is left open for the user to write and read records. '

8, 3.2 OPEN File

The purpose of the access OPEN function is to al/ow users la accessfiles that have

been previously created within the system. lf the file has already been created, all

a user wants to do is to read and write the records within the file. Thus the appropriate

acccs method call is to open the file for processing. _1 »

On the other hand, some high~level languages do not provide the capability of

allowing a user to create a new iile.,Instead, they allow the user to open a tile. lf

the file does not exist, it is assumed to be a user request to create a new file.

ln any case, the OPEN request formally connects the user’s program with a

file, The information required to OPEN a file can be specilied as followsz'

~ Device/user name or pathname

° Filename

ln addition, users are allowed to open _the file under certain user-defined cpn~

straints. Specifically, users can request the following: '”'_

~ OPEN for input only (i.e., the user will perform only READ type operations)

° OPEN for oulpul only (i.e,, the user will only WRITE to this ftle)

° OPEN for input and output (i.e., the user wants to be able to perform all types

of record operations on this file) _ V »

it is again up to the access methods to ensure that only those record requests

that conform to the mode under which the File was opened are actually processed.

All other requests must be rejected as errors. _

All other attributes of the File 'were specified at file CREATE time. Thus, with

the complete pathname to locate the file, the file system can then retrieve the me

attributes that were written at the time the file was built.

All the information written out as part of the CREATE processing is read back

as part of the OPEN processing. This is because the OPEN task must bring the file

system and access method back up to the level that existed just after the file had been

created. Therefore, the processing of the OPEN operation is as follows:

I. Validate Input Parameters. This essentially corresponds to the same step

in the CREATE processing. However, the input parameters are somewhat different

since we no longer need to specify all the file attributes: we need only find and read

them page How can we find the location of the file. and then its attributes? All that
s required is the file, or more specifically, the pathname, With these data, the file

system can read the directory entries until it finds a matching name, lt can then read

th¢ amibuttgg into memory, where the access method can copy them into the pre-

allocated file control blocks. '

Thus the only parameter that needs to be validated is the lile's pathname. lf

that is correct, the Gle can be found, and all the other parameters or attributes can

be read from the file itself. This is a critical point. We could have let the user rcspccify

all ol' the f`|le"s attributes. However, it is the job of the acccssfmethods to guarantee

that data are never datroyed within the file and that the records can always bc success-

fully read and written, barring hardware problems.

ll' the user must specify the lile's attributes each time, what if thc user makes

a mistake? What if there are 200 users each wanting access to the satne file? What

is the probability that alliof those 200 users will supply the file attribute data with

no errors? lt is simply not worth the risk to the data within the tile. The tile system

and the access methods should only trust the person who had the responsibility of

creating thc file and delining all of the |ile's attributes, lf the person who created

the file was wrong on one or more of the attributes, there is nothing that can be done,

short of recreating thelile. But if there is a conflict between the attributes recorded

in the file and the attributs passed by a user of the file, there is no choice but to

take the attributes that were written into the Ele itself. Also, remember, that all records

“Titten into the File were written according to the attributes defined within the file,

so they really cannot be changed on the fly.

2. Call the Basic File S ystem '.\‘ 0PEN_FILE Function. As part of the OPEN

processing. the access method must ensure that the lile is ready for the user to access

the records in it. l-`|rst, the access method must check the volume label to ensure that

the correct volume has been mounted on the device. Remember that file “abc” can

reside on multiple volumes (e.g., backup versions). Thus it is critical that the correct
volume be mounted and verified by the access method. Then, and only then, can

processing continue. _ Y '

Next, the access method must determine the size of the blocks in the file from

the file attribuies;'I'hen it must tell theiilesystctn toallocateone or more I/O buf-

fers large enough to contain the largest block in the file. Once this has been done,

I/O can be done by the lile system to read or write any block in the lile on behalf

of the user. _

The next function that must be performed is to allocate mainmemory space for

the FCB. Then the access method must copy the relevent file attributm into thc FCB

for future use V

Each of these functions is performed by calling the OPEN_Fl LE routine in the

basic tile system. Like thc CREATE processing, a fair amount of the processing

overlaps with that of the basic file system processing. '

3. Perfrom File-0rganization-spesific Processing. Although the OPEN

process is much more standard-across the set of tile organizations, there are still certain
unique things which must be done at OPEN time for particular file types. It
is the purpose of this section to highlight all file organization specific processing which

it uw; and above the standard processing. Typically. at the completion of OPEN

processing, the access methods set the current record pointer to point to the first user

data record in the file. This is done so that the user can immediately execute a READ

or RE.-\D_NEXT operation and have it succeed.

4. Read the File's Attributes. Now that the control blocks and E/O buffers

have been allocated, it is necessary to access all of the attributes of the Ele. These

characteristics can be read both from the lile’S directory entry and from the file iv

self. These attributes are then copied into the various control blocks within the tile

system.

5. Return Status and the File Identifier to the Calling Program. Again, as

with the CREATE request, the calling program must be informed as to whether or

not the request was completed successfully. lf it was, the user can assume that reading

and writing of the records can begin immediately. If there was an error, the user must

determine the next 'course of action. In any case, it is up to the user to determine

what to do next.

An OPEN function call can fail for a variety of reasons, such as:

* File cannot be found in the location specified by its pathname.

° File was found, but the current user does not have sufficient access rights to

use the file. _ V

° Unrecoverable hardware problems were encountered.

Also, the OPEN function returns to the calling program a file identifier, which

uniquely identifies this specific File among all the files that are currently OPEN on

the system. '

ln summary, an OPEN request is very similar to the basic file system‘s OPEN_

FILE function. The access method OPEN can be viewed as follows:

AM OPEN = (BFS OPEN_FlLE) + (AM-specific processing)

As was the case with the CREATE function, if the OPEN ,is successful, the

file is left in a state in which the user can then access the data records.

8. 3, 3 Close File

The purpose of the CLOSE function is Io allow users the opportunity y to deaccess,

or slap using, a particular file. After a user has processed the records in the file,

there needs to be some way to tell the access method “l’m done." This is one of

the tasks of the CLOSE request. When this function is invoked, the access method

knows that its sole responsibility is to ensure that the file is deceased correctly in

Such a way that no records are lost or destroyed. The only input required is the unique

file identifier that was returned by the file system by the CREATE or OPEN functions. To
accomplish this task, the CLOSE routine performs the following tasks
1. Write Out Any Remaining Data Still in I/0 Buffers. Sometimes, as we

shall soon see, the records and blocks in a file are not actually written back to the

file until they have to be. This cuts down on the actual number of real I/O opera-

tions performed and therefore improves performance. On the other hand, it then

forces on the CLOSE routine the responsibility to ensure that any data not yet writ-

ten into the file actually get transferred there. ln this manner, no records will be lost

and performance will be improved.

To perform this function, the access methods must maintain enough internal

control information to know the following;

~ What I/O buffers are in main memory?

~ Who owns the data in the l/O buffers?

° Which I/O buffers have been modified? x

2. Perform .File-Organization-Speci/ic Processingl This step mainly involves

ensuring that all critical file-processing information has been written out into thc file.

Although this may _be the case in most situations, there will arise cases in which the

data are not rewritten until the file is closed in order, for example, to improve per-

formance. Trailer labels on tape could be written out now, for example.

An access method might need to write out the following kinds of information:

° File-processing statistics '

° Current end»of-lile or last block used

° Current physical end-of-file r

By saving _these data, each access method will then be able to successfuly re-

OPEN the file later and correctly process all of its data records. ` i

3. _Call the Basic File System CLOSE_FILE Function. The purpose of this

call is to allow the basic file system to clean up its tables, as well as to be able to

deallocate any main memory that was used in support of theprocessing of that tile.

Also, the basic tile system will write out all blocks in I/O buffers which have been
modified since the last write of the buffer. ~

4. Return Status to the Calling Program. This is done for the same reasons

as mentioned previously. lt is brought up here again simply to help drive home th:

importance of notifying the calling program of the status (i.e., success or failure)

of the completed request.

It is possible for a CLOSE function to encounter errors and thus not complete

successfully. Typically, these kinds of errors are easy to fix, such as:

~ File to be CLOSEd has never been OPENed.

° File l/O errors encountered when writing the IlO buffers out into the lile.

~ Not _enough space on the medium to write out all of the modified l/O block

buffers.
ln summary, the CLOSE operation is almost the exact inverse of the processing that is performed at
OPEN time. Also, in is again wonhwhile 10 View the (1055

function' in :hc following PCISPCCUVCSI _

,s__\1 CLQSE = (BFS CLOSE_FlLE) + (AM~$pecific processing)

lr is uorrh noxing thai if a user fails lo issue the CLOSE requesr before finish-

ing cxccuiion of ihe program, it is the responsibility olflhe access method to perform

the proper processing as 1] the user had _vpecijically requested a close operalion lo

he performed. This is needed as a safeguard against potentially losing data in the

file. Also, ihcrc could have been a bug in the uscr’s program, but the user is not

yer aware ihar a problem exists. Remember, the file system and access method must

always work iogelher to ensure that the correct processing gels done, whether or not

the user program is “corrcct." _ -

I Ire purpose aj lhe ar.‘€&5 merlwu 1;/\11;1vu_;|re,jJum.'rrun 1; to allow user prvgrulm

lo increase lhe size of a file. This is essentially the same function as that provided

by the basic Iile system EXTEND_FlLE function. It is redone here to present to users

a consistent access method interface. *sf

The inputs to the EXTEND file function are as follows: '

° Unique lile identifier ` ~

' Size by which lilc is to be extended

° Contiguous allocation required, or not?

The processing that the EXTEND function performs is similar to the process»

ing done by the other access methodjile operations., -

I. Validate All Input Parameters. First, all input parameters are validated.

2. Call the Basic HI: Sys1em EX TENDTunction. Then, if all is well, the basic

File system EXTEND_FlLE funuion is called to physically extend the file by the

amount requested by the user program. -

3. l’crfur/n File~0rg`anizarion-Spec(/ic Processing. 'While the processing re-

quired for each lilc organization that is unique is minimal, there will becases in which

the EXTEND processing will be access method dependent. For example,-with the

more complex file organizations such as VSAM, space might be allocated for very

specific uses within the File itself, as opposed simply to adding to a pool of space

usable anywhere in the file structure.

4. Return Status lo the Calling Program. Finally, status is returned to the

calling program to indicate whether or not the file was extended successfully, As is

usually the case. it is possible for the EXTEND function not to complete processing

correetlv_ Some of thc reasons for this conclusion are as follows:

* Not enough total space available on the device. _ '

° if contiguous was requested, there might not be enough contiguous space

available on the device. - ' T

° Somesystems have a preset maximum number of times in which to EXTEND

the lile. This is typically done to prevent the case ofa runaway program taking

more and more room on a disk.

Thus the access method EXTEND processing can be summarized as follows:

AM EXTEND = (BFS EXTE_ND__FlI_E) + (AM-specific processing)