\documentstyle[11pt]{article}

\title{

Interactively Testing Remote Servers Using the Python Programming Language

}

\author{

Guido van Rossum \\

CWI, dept. CST; Kruislaan 413 \\

1098 SJ Amsterdam, The Netherlands \\

E-mail: {\tt [email protected]}

\and

Jelke de Boer \\

HIO Enschede; P.O.Box 1326 \\

7500 BH Enschede, The Netherlands

}

\begin{document}

\maketitle

\begin{abstract}

This paper describes how two tools that were developed quite

independently gained in power by a well-designed connection between

them. The tools are Python, an interpreted prototyping language, and

AIL, a Remote Procedure Call stub generator. The context is Amoeba, a

well-known distributed operating system developed jointly by the Free

University and CWI in Amsterdam.

As a consequence of their integration, both tools have profited:

Python gained usability when used with Amoeba --- for which it was not

specifically developed --- and AIL users now have a powerful

interactive tool to test servers and to experiment with new

client/server interfaces.%

\footnote{

An earlier version of this paper was presented at the Spring 1991

EurOpen Conference in Troms{\o} under the title ``Linking a Stub

Generator (AIL) to a Prototyping Language (Python).''

}

\end{abstract}

\section{Introduction}

Remote Procedure Call (RPC) interfaces, used in distributed systems

like Amoeba

\cite{Amoeba:IEEE,Amoeba:CACM},

have a much more concrete character than local procedure call

interfaces in traditional systems. Because clients and servers may

run on different machines, with possibly different word size, byte

order, etc., much care is needed to describe interfaces exactly and to

implement them in such a way that they continue to work when a client

or server is moved to a different machine. Since machines may fail

independently, error handling must also be treated more carefully.

A common approach to such problems is to use a {\em stub generator}.

This is a program that takes an interface description and transforms

it into functions that must be compiled and linked with client and

server applications. These functions are called by the application

code to take care of details of interfacing to the system's RPC layer,

to implement transformations between data representations of different

machines, to check for errors, etc. They are called `stubs' because

they don't actually perform the action that they are called for but

only relay the parameters to the server

\cite{RPC}.

Amoeba's stub generator is called AIL, which stands for Amoeba

Interface Language

\cite{AIL}.

The first version of AIL generated only C functions, but an explicit

goal of AIL's design was {\em retargetability}: it should be possible

to add back-ends that generate stubs for different languages from the

same interface descriptions. Moreover, the stubs generated by

different back-ends must be {\em interoperable}: a client written in

Modula-3, say, should be able to use a server written in C, and vice

versa.

This interoperability is the key to the success of the marriage

between AIL and Python. Python is a versatile interpreted language

developed by the first author. Originally intended as an alternative

for the kind of odd jobs that are traditionally solved by a mixture of

shell scripts, manually given shell commands, and an occasional ad hoc

C program, Python has evolved into a general interactive prototyping

language. It has been applied to a wide range of problems, from

replacements for large shell scripts to fancy graphics demos and

multimedia applications.

One of Python's strengths is the ability for the user to type in some

code and immediately run it: no compilation or linking is necessary.

Interactive performance is further enhanced by Python's concise, clear

syntax, its very-high-level data types, and its lack of declarations

(which is compensated by run-time type checking). All this makes

programming in Python feel like a leisure trip compared to the hard

work involved in writing and debugging even a smallish C program.

It should be clear by now that Python will be the ideal tool to test

servers and their interfaces. Especially during the development of a

complex server, one often needs to generate test requests on an ad hoc

basis, to answer questions like ``what happens if request X arrives

when the server is in state Y,'' to test the behavior of the server

with requests that touch its limitations, to check server responses to

all sorts of wrong requests, etc. Python's ability to immediately

execute `improvised' code makes it a much better tool for this

situation than C.

The link to AIL extends Python with the necessary functionality to

connect to arbitrary servers, making the server testbed sketched above

a reality. Python's high-level data types, general programming

features, and system interface ensure that it has all the power and

flexibility needed for the job.

One could go even further than this. Current distributed operating

systems, based on client-server interaction, all lack a good command

language or `shell' to give adequate access to available services.

Python has considerable potential for becoming such a shell.

\subsection{Overview of this Paper}

The rest of this paper contains three major sections and a conclusion.

First an overview of the Python programming language is given. Next

comes a short description of AIL, together with some relevant details

about Amoeba. Finally, the design and construction of the link

between Python and AIL is described in much detail. The conclusion

looks back at the work and points out weaknesses and strengths of

Python and AIL that were discovered in the process.

\section{An Overview of Python}

Python%

\footnote{

Named after the funny TV show, not the nasty reptile.

}

owes much to ABC

\cite{ABC},

a language developed at CWI as a programming language for non-expert

computer users. Python borrows freely from ABC's syntax and data

types, but adds modules, exceptions and classes, extensibility, and

the ability to call system functions. The concepts of modules,

exceptions and (to some extent) classes are influenced strongly by

their occurrence in Modula-3

\cite{Modula-3}.

Although Python resembles ABC in many ways, there is a a clear

difference in application domain. ABC is intended to be the only

programming language for those who use a computer as a tool, but

occasionally need to write a program. For this reason, ABC is not

just a programming language but also a programming environment, which

comes with an integrated syntax-directed editor and some source

manipulation commands. Python, on the other hand, aims to be a tool

for professional (system) programmers, for whom having a choice of

languages with different feature sets makes it possible to choose `the

right tool for the job.' The features added to Python make it more

useful than ABC in an environment where access to system functions

(such as file and directory manipulations) are common. They also

support the building of larger systems and libraries. The Python

implementation offers little in the way of a programming environment,

but is designed to integrate seamlessly with existing programming

environments (e.g. UNIX and Emacs).

Perhaps the best introduction to Python is a short example. The

following is a complete Python program to list the contents of a UNIX

directory.

\begin{verbatim}

import sys, posix

def ls(dirname): # Print sorted directory contents

names = posix.listdir(dirname)

names.sort()

for name in names:

if name[0] != '.': print name

ls(sys.argv[1])

\end{verbatim}

The largest part of this program, in the middle starting with {\tt

def}, is a function definition. It defines a function named {\tt ls}

with a single parameter called {\tt dirname}. (Comments in Python

start with `\#' and extend to the end of the line.) The function body

is indented: Python uses indentation for statement grouping instead of

braces or begin/end keywords. This is shorter to type and avoids

frustrating mismatches between the perception of grouping by the user

and the parser. Python accepts one statement per line; long

statements may be broken in pieces using the standard backslash

convention. If the body of a compound statement is a single, simple

statement, it may be placed on the same line as the head.

The first statement of the function body calls the function {\tt

listdir} defined in the module {\tt posix}. This function returns a

list of strings representing the contents of the directory name passed

as a string argument, here the argument {\tt dirname}. If {\tt

dirname} were not a valid directory name, or perhaps not even a

string, {\tt listdir} would raise an exception and the next statement

would never be reached. (Exceptions can be caught in Python; see

later.) Assuming {\tt listdir} returns normally, its result is

assigned to the local variable {\tt names}.

The second statement calls the method {\tt sort} of the variable {\tt

names}. This method is defined for all lists in Python and does the

obvious thing: the elements of the list are reordered according to

their natural ordering relationship. Since in our example the list

contains strings, they are sorted in ascending ASCII order.

The last two lines of the function contain a loop that prints all

elements of the list whose first character isn't a period. In each

iteration, the {\tt for} statement assigns an element of the list to

the local variable {\tt name}. The {\tt print} statement is intended

for simple-minded output; more elaborate formatting is possible with

Python's string handling functions.

The other two parts of the program are easily explained. The first

line is an {\tt import} statement that tells the interpreter to import

the modules {\tt sys} and {\tt posix}. As it happens these are both

built into the interpreter. Importing a module (built-in or

otherwise) only makes the module name available in the current scope;

functions and data defined in the module are accessed through the dot

notation as in {\tt posix.listdir}. The scope rules of Python are

such that the imported module name {\tt posix} is also available in

the function {\tt ls} (this will be discussed in more detail later).

Finally, the last line of the program calls the {\tt ls} function with

a definite argument. It must be last since Python objects must be

defined before they can be used; in particular, the function {\tt ls}

must be defined before it can be called. The argument to {\tt ls} is

{\tt sys.argv[1]}, which happens to be the Python equivalent of {\tt

\$1} in a shell script or {\tt argv[1]} in a C program's {\tt main}

function.

\subsection{Python Data Types}

(This and the following subsections describe Python in quite a lot of

detail. If you are more interested in AIL, Amoeba and how they are

linked with Python, you can skip to section 3 now.)

Python's syntax may not have big surprises (which is exactly as it

should be), but its data types are quite different from what is found

in languages like C, Ada or Modula-3. All data types in Python, even

integers, are `objects'. All objects participate in a common garbage

collection scheme (currently implemented using reference counting).

Assignment is cheap, independent of object size and type: only a

pointer to the assigned object is stored in the assigned-to variable.

No type checking is performed on assignment; only specific operations

like addition test for particular operand types.

The basic object types in Python are numbers, strings, tuples, lists

and dictionaries. Some other object types are open files, functions,

modules, classes, and class instances; even types themselves are

represented as objects. Extension modules written in C can define

additional object types; examples are objects representing windows and

Amoeba capabilities. Finally, the implementation itself makes heavy

use of objects, and defines some private object types that aren't

normally visible to the user. There is no explicit pointer type in

Python.

{\em Numbers}, both integers and floating point, are pretty

straightforward. The notation for numeric literals is the same as in

C, including octal and hexadecimal integers; precision is the same as

{\tt long} or {\tt double} in C\@. A third numeric type, `long

integer', written with an `L' suffix, can be used for arbitrary

precision calculations. All arithmetic, shifting and masking

operations from C are supported.

{\em Strings} are `primitive' objects just like numbers. String

literals are written between single quotes, using similar escape

sequences as in C\@. Operations are built into the language to

concatenate and to replicate strings, to extract substrings, etc.

There is no limit to the length of the strings created by a program.

There is no separate character data type; strings of length one do

nicely.

{\em Tuples} are a way to `pack' small amounts of heterogeneous data

together and carry them around as a unit. Unlike structure members in

C, tuple items are nameless. Packing and unpacking assignments allow

access to the items, for example:

\begin{verbatim}

x = 'Hi', (1, 2), 'World' # x is a 3-item tuple,

# its middle item is (1, 2)

p, q, r = x # unpack x into p, q and r

a, b = q # unpack q into a and b

\end{verbatim}

A combination of packing and unpacking assignment can be used as

parallel assignment, and is idiom for permutations, e.g.:

\begin{verbatim}

p, q = q, p # swap without temporary

a, b, c = b, c, a # cyclic permutation

\end{verbatim}

Tuples are also used for function argument lists if there is more than

one argument. A tuple object, once created, cannot be modified; but

it is easy enough to unpack it and create a new, modified tuple from

the unpacked items and assign this to the variable that held the

original tuple object (which will then be garbage-collected).

{\em Lists} are array-like objects. List items may be arbitrary

objects and can be accessed and changed using standard subscription

notation. Lists support item insertion and deletion, and can

therefore be used as queues, stacks etc.; there is no limit to their

size.

Strings, tuples and lists together are {\em sequence} types. These

share a common notation for generic operations on sequences such as

subscription, concatenation, slicing (taking subsequences) and

membership tests. As in C, subscripts start at 0.

{\em Dictionaries} are `mappings' from one domain to another. The

basic operations on dictionaries are item insertion, extraction and

deletion, using subscript notation with the key as subscript. (The

current implementation allows only strings in the key domain, but a

future version of the language may remove this restriction.)

\subsection{Statements}

Python has various kinds of simple statements, such as assignments

and {\tt print} statements, and several kinds of compound statements,

like {\tt if} and {\tt for} statements. Formally, function

definitions and {\tt import} statements are also statements, and there

are no restrictions on the ordering of statements or their nesting:

{\tt import} may be used inside a function, functions may be defined

conditionally using an {\tt if} statement, etc. The effect of a

declaration-like statement takes place only when it is executed.

All statements except assignments and expression statements begin with

a keyword: this makes the language easy to parse. An overview of the

most common statement forms in Python follows.

An {\em assignment} has the general form

\vspace{\itemsep}

\noindent

{\em variable $=$ variable $= ... =$ variable $=$ expression}

\vspace{\itemsep}

It assigns the value of the expression to all listed variables. (As

shown in the section on tuples, variables and expressions can in fact

be comma-separated lists.) The assignment operator is not an

expression operator; there are no horrible things in Python like

\begin{verbatim}

while (p = p->next) { ... }

\end{verbatim}

Expression syntax is mostly straightforward and will not be explained

in detail here.

An {\em expression statement} is just an expression on a line by

itself. This writes the value of the expression to standard output,

in a suitably unambiguous way, unless it is a `procedure call' (a

function call that returns no value). Writing the value is useful

when Python is used in `calculator mode', and reminds the programmer

not to ignore function results.

The {\tt if} statement allows conditional execution. It has optional

{\tt elif} and {\tt else} parts; a construct like {\tt

if...elif...elif...elif...else} can be used to compensate for the

absence of a {\em switch} or {\em case} statement.

Looping is done with {\tt while} and {\tt for} statements. The latter

(demonstrated in the `ls' example earlier) iterates over the elements

of a `sequence' (see the discussion of data types below). It is

possible to terminate a loop with a {\tt break} statement or to start

the next iteration with {\tt continue}. Both looping statements have

an optional {\tt else} clause which is executed after the loop is

terminated normally, but skipped when it is terminated by {\tt break}.

This can be handy for searches, to handle the case that the item is

not found.

Python's {\em exception} mechanism is modelled after that of Modula-3.

Exceptions are raised by the interpreter when an illegal operation is

tried. It is also possible to explicitly raise an exception with the

{\tt raise} statement:

\vspace{\itemsep}

\noindent

{\tt raise {\em expression}, {\em expression}}

\vspace{\itemsep}

The first expression identifies which exception should be raised;

there are several built-in exceptions and the user may define

additional ones. The second, optional expression is passed to the

handler, e.g. as a detailed error message.

Exceptions may be handled (caught) with the {\tt try} statement, which

has the following general form:

\vspace{\itemsep}

\noindent

{\tt

\begin{tabular}{l}

try: {\em block} \\

except {\em expression}, {\em variable}: {\em block} \\

except {\em expression}, {\em variable}: {\em block} \\

... \\

except: {\em block}

\end{tabular}

}

\vspace{\itemsep}

When an exception is raised during execution of the first block, a

search for an exception handler starts. The first {\tt except} clause

whose {\em expression} matches the exception is executed. The

expression may specify a list of exceptions to match against. A

handler without an expression serves as a `catch-all'. If there is no

match, the search for a handler continues with outer {\tt try}

statements; if no match is found on the entire invocation stack, an

error message and stack trace are printed, and the program is

terminated (interactively, the interpreter returns to its main loop).

Note that the form of the {\tt except} clauses encourages a style of

programming whereby only selected exceptions are caught, passing

unanticipated exceptions on to the caller and ultimately to the user.

This is preferable over a simpler `catch-all' error handling

mechanism, where a simplistic handler intended to catch a single type

of error like `file not found' can easily mask genuine programming

errors --- especially in a language like Python which relies strongly

on run-time checking and allows the catching of almost any type of

error.

Other common statement forms, which we have already encountered, are

function definitions, {\tt import} statements and {\tt print}

statements. There is also a {\tt del} statement to delete one or more

variables, a {\tt return} statement to return from a function, and a

{\tt global} statement to allow assignments to global variables.

Finally, the {\tt pass} statement is a no-op.

\subsection{Execution Model}

A Python program is executed by a stack-based interpreter.

When a function is called, a new `execution environment' for it is

pushed onto the stack. An execution environment contains (among other

data) pointers to two `symbol tables' that are used to hold variables:

the local and the global symbol table. The local symbol table

contains local variables of the current function invocation (including

the function arguments); the global symbol table contains variables

defined in the module containing the current function.

The `global' symbol table is thus only global with respect to the

current function. There are no system-wide global variables; using

the {\tt import} statement it is easy enough to reference variables

that are defined in other modules. A system-wide read-only symbol

table is used for built-in functions and constants though.

On assignment to a variable, by default an entry for it is made in the

local symbol table of the current execution environment. The {\tt

global} command can override this (it is not enough that a global

variable by the same name already exists). When a variable's value is

needed, it is searched first in the local symbol table, then in the

global one, and finally in the symbol table containing built-in

functions and constants.

The term `variable' in this context refers to any name: functions and

imported modules are searched in exactly the same way.

Names defined in a module's symbol table survive until the end of the

program. This approximates the semantics of file-static global

variables in C or module variables in Modula-3. A module is

initialized the first time it is imported, by executing the text of

the module as a parameterless function whose local and global symbol

tables are the same, so names are defined in module's symbol table.

(Modules implemented in C have another way to define symbols.)

A Python main program is read from standard input or from a script

file passed as an argument to the interpreter. It is executed as if

an anonymous module was imported. Since {\tt import} statements are

executed like all other statements, the initialization order of the

modules used in a program is defined by the flow of control through

the program.

The `attribute' notation {\em m.name}, where {\em m} is a module,

accesses the symbol {\em name} in that module's symbol table. It can

be assigned to as well. This is in fact a special case of the

construct {\em x.name} where {\em x} denotes an arbitrary object; the

type of {\em x} determines how this is to be interpreted, and what

assignment to it means.

For instance, when {\tt a} is a list object, {\tt a.append} yields a

built-in `method' object which, when called, appends an item to {\tt a}.

(If {\tt a} and {\tt b} are distinct list objects, {\tt a.append} and

{\tt b.append} are distinguishable method objects.) Normally, in

statements like {\tt a.append(x)}, the method object {\tt a.append} is

called and then discarded, but this is a matter of convention.

List attributes are read-only --- the user cannot define new list

methods. Some objects, like numbers and strings, have no attributes

at all. Like all type checking in Python, the meaning of an attribute

is determined at run-time --- when the parser sees {\em x.name}, it

has no idea of the type of {\em x}. Note that {\em x} here does not

have to be a variable --- it can be an arbitrary (perhaps

parenthesized) expression.

Given the flexibility of the attribute notation, one is tempted to use

methods to replace all standard operations. Yet, Python has kept a

small repertoire of built-in functions like {\tt len()} and {\tt

abs()}. The reason is that in some cases the function notation is

more familiar than the method notation; just like programs would

become less readable if all infix operators were replaced by function

calls, they would become less readable if all function calls had to be

replaced by method calls (and vice versa!).

The choice whether to make something a built-in function or a method

is a matter of taste. For arithmetic and string operations, function

notation is preferred, since frequently the argument to such an

operation is an expression using infix notation, as in {\tt abs(a+b)};

this definitely looks better than {\tt (a+b).abs()}. The choice

between make something a built-in function or a function defined in a

built-in method (requiring {\tt import}) is similarly guided by

intuition; all in all, only functions needed by `general' programming

techniques are built-in functions.

\subsection{Classes}

Python has a class mechanism distinct from the object-orientation

already explained. A class in Python is not much more than a

collection of methods and a way to create class instances. Class

methods are ordinary functions whose first parameter is the class

instance; they are called using the method notation.

For instance, a class can be defined as follows:

\begin{verbatim}

class Foo:

def meth1(self, arg): ...

def meth2(self): ...

\end{verbatim}

A class instance is created by

{\tt x = Foo()}

and its methods can be called thus:

\begin{verbatim}

x.meth1('Hi There!')

x.meth2()

\end{verbatim}

The functions used as methods are also available as attributes of the

class object, and the above method calls could also have been written

as follows:

\begin{verbatim}

Foo.meth1(x, 'Hi There!')

Foo.meth2(x)

\end{verbatim}

Class methods can store instance data by assigning to instance data

attributes, e.g.:

\begin{verbatim}

self.size = 100

self.title = 'Dear John'

\end{verbatim}

Data attributes do not have to be declared; as with local variables,

they spring into existence when assigned to. It is a matter of

discretion to avoid name conflicts with method names. This facility

is also available to class users; instances of a method-less class can

be used as records with named fields.

There is no built-in mechanism for instance initialization. Classes

by convention provide an {\tt init()} method which initializes the

instance and then returns it, so the user can write

\begin{verbatim}

x = Foo().init('Dr. Strangelove')

\end{verbatim}

Any user-defined class can be used as a base class to derive other

classes. However, built-in types like lists cannot be used as base

classes. (Incidentally, the same is true in C++ and Modula-3.) A

class may override any method of its base classes. Instance methods

are first searched in the method list of their class, and then,

recursively, in the method lists of their base class. Initialization

methods of derived classes should explicitly call the initialization

methods of their base class.

A simple form of multiple inheritance is also supported: a class can

have multiple base classes, but the language rules for resolving name

conflicts are somewhat simplistic, and consequently the feature has so

far found little usage.

\subsection{The Python Library}

Python comes with an extensive library, structured as a collection of

modules. A few modules are built into the interpreter: these

generally provide access to system libraries implemented in C such as

mathematical functions or operating system calls. Two built-in

modules provide access to internals of the interpreter and its

environment. Even abusing these internals will at most cause an

exception in the Python program; the interpreter will not dump core

because of errors in Python code.

Most modules however are written in Python and distributed with the

interpreter; they provide general programming tools like string

operations and random number generators, provide more convenient

interfaces to some built-in modules, or provide specialized services

like a {\em getopt}-style command line option processor for

stand-alone scripts.

There are also some modules written in Python that dig deep in the

internals of the interpreter; there is a module to browse the stack

backtrace when an unhandled exception has occurred, one to disassemble

the internal representation of Python code, and even an interactive

source code debugger which can trace Python code, set breakpoints,

etc.

\subsection{Extensibility}

It is easy to add new built-in modules written in C to the Python

interpreter. Extensions appear to the Python user as built-in

modules. Using a built-in module is no different from using a module

written in Python, but obviously the author of a built-in module can

do things that cannot be implemented purely in Python.

In particular, built-in modules can contain Python-callable functions

that call functions from particular system libraries (`wrapper

functions'), and they can define new object types. In general, if a

built-in module defines a new object type, it should also provide at

least one function that creates such objects. Attributes of such

object types are also implemented in C; they can return data

associated with the object or methods, implemented as C functions.

For instance, an extension was created for Amoeba: it provides wrapper

functions for the basic Amoeba name server functions, and defines a

`capability' object type, whose methods are file server operations.

Another extension is a built-in module called {\tt posix}; it provides

wrappers around post UNIX system calls. Extension modules also

provide access to two different windowing/graphics interfaces: STDWIN

\cite{STDWIN}

(which connects to X11 on UNIX and to the Mac Toolbox on the

Macintosh), and the Graphics Library (GL) for Silicon Graphics

machines.

Any function in an extension module is supposed to type-check its

arguments; the interpreter contains a convenience function to

facilitate extracting C values from arguments and type-checking them

at the same time. Returning values is also painless, using standard

functions to create Python objects from C values.

On some systems extension modules may be dynamically loaded, thus

avoiding the need to maintain a private copy of the Python interpreter

in order to use a private extension.

\section{A Short Description of AIL and Amoeba}

An RPC stub generator takes an interface description as input. The

designer of a stub generator has at least two choices for the input

language: use a suitably restricted version of the target language, or

design a new language. The first solution was chosen, for instance,

by the designers of Flume, the stub generator for the Topaz

distributed operating system built at DEC SRC

\cite{Flume,Evolving}.

Flume's one and only target language is Modula-2+ (the predecessor of

Modula-3). Modula-2+, like Modula-N for any N, has an interface

syntax that is well suited as a stub generator input language: an

interface module declares the functions that are `exported' by a

module implementation, with their parameter and return types, plus the

types and constants used for the parameters. Therefore, the input to

Flume is simply a Modula-2+ interface module. But even in this ideal

situation, an RPC stub generator needs to know things about functions

that are not stated explicitly in the interface module: for instance,

the transfer direction of VAR parameters (IN, OUT or both) is not

given. Flume solves this and other problems by a mixture of

directives hidden in comments and a convention for the names of

objects. Thus, one could say that the designers of Flume really

created a new language, even though it looks remarkably like their

target language.

\subsection{The AIL Input Language}

Amoeba uses C as its primary programming language. C function

declarations (at least in `Classic' C) don't specify the types of

the parameters, let alone their transfer direction. Using this as

input for a stub generator would require almost all information for

the stub generator to be hidden inside comments, which would require a

rather contorted scanner. Therefore we decided to design the input

syntax for Amoeba's stub generator `from scratch'. This gave us the

liberty to invent proper syntax not only for the transfer direction of

parameters, but also for variable-length arrays.

On the other hand we decided not to abuse our freedom, and borrowed as

much from C as we could. For instance, AIL runs its input through the

C preprocessor, so we get macros, include files and conditional

compilation for free. AIL's type declaration syntax is a superset of

C's, so the user can include C header files to use the types declared

there as function parameter types --- which are declared using

function prototypes as in C++ or Standard C\@. It should be clear by

now that AIL's lexical conventions are also identical to C's. The

same is true for its expression syntax.

Where does AIL differ from C, then? Function declarations in AIL are

grouped in {\em classes}. Classes in AIL are mostly intended as a

grouping mechanism: all functions implemented by a server are grouped

together in a class. Inheritance is used to form new groups by adding

elements to existing groups; multiple inheritance is supported to join

groups together. Classes can also contain constant and type

definitions, and one form of output that AIL can generate is a header

file for use by C programmers who wish to use functions from a

particular AIL class.

Let's have a look at some (unrealistically simple) class definitions:

\begin{verbatim}

#include <amoeba.h> /* Defines `capability', etc. */

class standard_ops [1000 .. 1999] {

/* Operations supported by most interfaces */

std_info(*, out char buf[size:100], out int size);

std_destroy(*);

};

\end{verbatim}

This defines a class called `standard\_ops' whose request codes are

chosen by AIL from the range 1000-1999. Request codes are small

integers used to identify remote operations. The author of the class

must specify a range from which AIL chooses, and class authors must

make sure they avoid conflicts, e.g. by using an `assigned number

administration office'. In the example, `std\_info' will be assigned

request code 1000 and `std\_destroy' will get code 1001. There is

also an option to explicitly assign request codes, for compatibility

with servers with manually written interfaces.

The class `standard\_ops' defines two operations, `std\_info' and

`std\_destroy'. The first parameter of each operation is a star

(`*'); this is a placeholder for a capability that must be passed when

the operation is called. The description of Amoeba below explains the

meaning and usage of capabilities; for now, it is sufficient to know

that a capability is a small structure that uniquely identifies an

object and a server or service.

The standard operation `std\_info' has two output parameters: a

variable-size character buffer (which will be filled with a short

descriptive string of the object to which the operation is applied)

and an integer giving the length of this string. The standard

operation `std\_destroy' has no further parameters --- it just

destroys the object, if the caller has the right to do so.

The next class is called `tty':

\begin{verbatim}

class tty [2000 .. 2099] {

inherit standard_ops;

const TTY_MAXBUF = 1000;

tty_write(*, char buf[size:TTY_MAXBUF], int size);

tty_read(*, out char buf[size:TTY_MAXBUF], out int size);

};

\end{verbatim}

The request codes for operations defined in this class lie in the

range 2000-2099; inherited operations use the request codes already

assigned to them. The operations defined by this class are

`tty\_read' and `tty\_write', which pass variable-sized data buffers

between client and server. Class `tty' inherits class

`standard\_ops', so tty objects also support the operations

`std\_info' and `std\_destroy'.

Only the {\em interface} for `std\_info' and `std\_destroy' is shared

between tty objects and other objects whose interface inherits

`standard\_ops'; the implementation may differ. Even multiple

implementations of the `tty' interface may exist, e.g. a driver for a

console terminal and a terminal emulator in a window. To expand on

the latter example, consider:

\begin{verbatim}

class window [2100 .. 2199] {

inherit standard_ops;

win_create(*, int x, int y, int width, int height,

out capability win_cap);

win_reconfigure(*, int x, int y, int width, int height);

};

class tty_emulator [2200 .. 2299] {

inherit tty, window;

};

\end{verbatim}

Here two new interface classes are defined.

Class `window' could be used for creating and manipulating windows.

Note that `win\_create' returns a capability for the new window.

This request should probably should be sent to a generic window

server capability, or it might create a subwindow when applied to a

window object.

Class `tty\_emulator' demonstrates the essence of multiple inheritance.

It is presumably the interface to a window-based terminal emulator.

Inheritance is transitive, so `tty\_emulator' also implicitly inherits

`standard\_ops'.

In fact, it inherits it twice: once via `tty' and once via `window'.

Since AIL class inheritance only means interface sharing, not

implementation sharing, inheriting the same class multiple times is

never a problem and has the same effect as inheriting it once.

Note that the power of AIL classes doesn't go as far as C++.

AIL classes cannot have data members, and there is

no mechanism for a server that implements a derived class

to inherit the implementation of the base

class --- other than copying the source code.

The syntax for class definitions and inheritance is also different.

\subsection{Amoeba}

The smell of `object-orientedness' that the use of classes in AIL

creates matches nicely with Amoeba's object-oriented approach to

RPC\@. In Amoeba, almost all operating system entities (files,

directories, processes, devices etc.) are implemented as {\em

objects}. Objects are managed by {\em services} and represented by

{\em capabilities}. A capability gives its holder access to the

object it represents. Capabilities are protected cryptographically

against forgery and can thus be kept in user space. A capability is a

128-bit binary string, subdivided as follows:

% XXX Need a better version of this picture!

\begin{verbatim}

48 24 8 48 Bits

+----------------+------------+--------+---------------+

| Service | Object | Perm. | Check |

| port | number | bits | word |

+----------------+------------+--------+---------------+

\end{verbatim}

The service port is used by the RPC implementation in the Amoeba

kernel to locate a server implementing the service that manages the

object. In many cases there is a one-to-one correspondence between

servers and services (each service is implemented by exactly one

server process), but some services are replicated. For instance,

Amoeba's directory service, which is crucial for gaining access to most

other services, is implemented by two servers that listen on the same

port and know about exactly the same objects.

The object number in the capability is used by the server receiving

the request for identifying the object to which the operation applies.

The permission bits specify which operations the holder of the capability

may apply. The last part of a capability is a 48-bit long `check

word', which is used to prevent forgery. The check word is computed

by the server based upon the permission bits and a random key per object

that it keeps secret. If you change the permission bits you must compute

the proper check word or else the server will refuse the capability.

Due to the size of the check word and the nature of the cryptographic

`one-way function' used to compute it, inverting this function is

impractical, so forging capabilities is impossible.%

\footnote{

As computers become faster, inverting the one-way function becomes

less impractical.

Therefore, a next version of Amoeba will have 64-bit check words.

}

A working Amoeba system is a collection of diverse servers, managing

files, directories, processes, devices etc. While most servers have

their own interface, there are some requests that make sense for some

or all object types. For instance, the {\em std\_info()} request,

which returns a short descriptive string, applies to all object types.

Likewise, {\em std\_destroy()} applies to files, directories and

processes, but not to devices.

Similarly, different file server implementations may want to offer the

same interface for operations like {\em read()} and {\em write()} to

their clients. AIL's grouping of requests into classes is ideally

suited to describe this kind of interface sharing, and a class

hierarchy results which clearly shows the similarities between server

interfaces (not necessarily their implementations!).

The base class of all classes defines the {\em std\_info()} request.

Most server interfaces actually inherit a derived class that also

defines {\em std\_destroy().} File servers inherit a class that

defines the common operations on files, etc.

\subsection{How AIL Works}

The AIL stub generator functions in three phases:

\begin{itemize}

\item

parsing,

\item

strategy determination,

\item

code generation.

\end{itemize}

{\bf Phase one} parses the input and builds a symbol table containing

everything it knows about the classes and other definitions found in

the input.

{\bf Phase two} determines the strategy to use for each function

declaration in turn and decides upon the request and reply message

formats. This is not a simple matter, because of various optimization

attempts. Amoeba's kernel interface for RPC requests takes a

fixed-size header and one arbitrary-size buffer. A large part of the

header holds the capability of the object to which the request is

directed, but there is some space left for a few integer parameters

whose interpretation is left up to the server. AIL tries to use these

slots for simple integer parameters, for two reasons.

First, unlike the buffer, header fields are byte-swapped by the RPC

layer in the kernel if necessary, so it saves a few byte swapping

instructions in the user code. Second, and more important, a common

form of request transfers a few integers and one large buffer to or

from a server. The {\em read()} and {\em write()} requests of most

file servers have this form, for instance. If it is possible to place

all integer parameters in the header, the address of the buffer

parameter can be passed directly to the kernel RPC layer. While AIL

is perfectly capable of handling requests that do not fit this format,

the resulting code involves allocating a new buffer and copying all

parameters into it. It is a top priority to avoid this copying

(`marshalling') if at all possible, in order to maintain Amoeba's

famous RPC performance.

When AIL resorts to copying parameters into a buffer, it reorders them

so that integers indicating the lengths of variable-size arrays are

placed in the buffer before the arrays they describe, since otherwise

decoding the request would be impossible. It also adds occasional

padding bytes to ensure integers are aligned properly in the buffer ---

this can speed up (un)marshalling.

{\bf Phase three} is the code generator, or back-end. There are in

fact many different back-ends that may be called in a single run to

generate different types of output. The most important output types

are header files (for inclusion by the clients of an interface),

client stubs, and `server main loop' code. The latter decodes

incoming requests in the server. The generated code depends on the

programming language requested, and there are separate back-ends for

each supported language.

It is important that the strategy chosen by phase two is independent

of the language requested for phase three --- otherwise the

interoperability of servers and clients written in different languages

would be compromised.

\section{Linking AIL to Python}

From the previous section it can be concluded that linking AIL to

Python is a matter of writing a back-end for Python. This is indeed

what we did.

Considerable time went into the design of the back-end in order to

make the resulting RPC interface for Python fit as smoothly as

possible in Python's programming style. For instance, the issues of

parameter transfer, variable-size arrays, error handling, and call

syntax were all solved in a manner that favors ease of use in Python

rather than strict correspondence with the stubs generated for C,

without compromising network-level compatibility.

\subsection{Mapping AIL Entities to Python}

For each programming language that AIL is to support, a mapping must

be designed between the data types in AIL and those in that language.

Other aspects of the programming languages, such as differences in

function call semantics, must also be taken care of.

While the mapping for C is mostly straightforward, the mapping for

Python requires a little thinking to get the best results for Python

programmers.

\subsubsection{Parameter Transfer Direction}

Perhaps the simplest issue is that of parameter transfer direction.

Parameters of functions declared in AIL are categorized as being of

type {\tt in}, {\tt out} or {\tt in} {\tt out} (the same distinction

as made in Ada). Python only has call-by-value parameter semantics;

functions can return multiple values as a tuple. This means that,

unlike the C back-end, the Python back-end cannot always generate

Python functions with exactly the same parameter list as the AIL

functions.

Instead, the Python parameter list consists of all {\tt in} and {\tt

in} {\tt out} parameters, in the order in which they occur in the AIL

parameter list; similarly, the Python function returns a tuple

containing all {\tt in} {\tt out} and {\tt out} parameters. In fact

Python packs function parameters into a tuple as well, stressing the

symmetry between parameters and return value. For example, a stub

with this AIL parameter list:

\begin{verbatim}

(*, in int p1, in out int p2, in int p3, out int p4)

\end{verbatim}

will have the following parameter list and return values in Python:

\begin{verbatim}

(p1, p2, p3) -> (p2, p4)

\end{verbatim}

\subsubsection{Variable-size Entities}

The support for variable-size objects in AIL is strongly guided by the

limitations of C in this matter. Basically, AIL allows what is

feasible in C: functions may have variable-size arrays as parameters

(both input or output), provided their length is passed separately.

In practice this is narrowed to the following rule: for each

variable-size array parameter, there must be an integer parameter

giving its length. (An exception for null-terminated strings is

planned but not yet realized.)

Variable-size arrays in AIL or C correspond to {\em sequences} in

Python: lists, tuples or strings. These are much easier to use than

their C counterparts. Given a sequence object in Python, it is always

possible to determine its size: the built-in function {\tt len()}

returns it. It would be annoying to require the caller of an RPC stub

with a variable-size parameter to also pass a parameter that

explicitly gives its size. Therefore we eliminate all parameters from

the Python parameter list whose value is used as the size of a

variable-size array. Such parameters are easily found: the array

bound expression contains the name of the parameter giving its size.

This requires the stub code to work harder (it has to recover the

value for size parameters from the corresponding sequence parameter),

but at least part of this work would otherwise be needed as well, to

check that the given and actual sizes match.

Because of the symmetry in Python between the parameter list and the

return value of a function, the same elimination is performed on

return values containing variable-size arrays: integers returned

solely to tell the client the size of a returned array are not

returned explicitly to the caller in Python.

\subsubsection{Error Handling}

Another point where Python is really better than C is the issue of

error handling. It is a fact of life that everything involving RPC

may fail, for a variety of reasons outside the user's control: the

network may be disconnected, the server may be down, etc. Clients

must be prepared to handle such failures and recover from them, or at

least print an error message and die. In C this means that every

function returns an error status that must be checked by the caller,

causing programs to be cluttered with error checks --- or worse,

programs that ignore errors and carry on working with garbage data.