This document describes how to write modules in C or C++ to extend the

Python interpreter. It also describes how to use Python as an

`embedded' language, and how extension modules can be loaded

dynamically (at run time) into the interpreter, if the operating

system supports this feature.

\section{Introduction}

programmer as \code{foo} is generally implemented by a file called

\file{foomodule.c}. All but the most essential standard built-in

modules also adhere to this convention, and in fact some of them form

excellent examples of how to create an extension.

Python: they can implement new data types, and they can make system

calls or call C library functions. Since the latter is usually the

most important reason for adding an extension, I'll concentrate on

adding `wrappers' around C library functions; the concrete example

uses the wrapper for

\code{system()} in module \code{posix}, found in (of course) the file

\file{posixmodule.c}.

`boilerplate' code (starting right with the copyright notice)!

Let's skip the boilerplate and have a look at an interesting function

in \file{posixmodule.c} first:

in Python (here the single value \code{'ls -l'}) to the arguments that

are passed to the C function. The C function always has two

parameters, conventionally named \var{self} and \var{args}. In this

example, \var{self} will always be a \code{NULL} pointer, since this is a

function, not a method (this is done so that the interpreter doesn't

have to understand two different types of C functions).

The \var{args} parameter will be a pointer to a Python object, or

\code{NULL} if the Python function/method was called without

arguments. It is necessary to do full argument type checking on each

call, since otherwise the Python user would be able to cause the

Python interpreter to `dump core' by passing the wrong arguments to a

function in an extension module (or no arguments at all). Because

argument checking and converting arguments to C is such a common task,

there's a general function in the Python interpreter which combines

these tasks: \code{getargs()}. It uses a template string to determine

both the types of the Python argument and the types of the C variables

into which it should store the converted values. (More about this

later.)\footnote{

There are convenience macros \code{getstrarg()},

\code{getintarg()}, etc., for many common forms of \code{getargs()}

templates. These are relics from the past; it's better to call

\code{getargs()} directly.}

If \code{getargs()} returns nonzero, the argument list has the right

type and its components have been stored in the variables whose

addresses are passed. If it returns zero, an error has occurred. In

the latter case it has already raised an appropriate exception by

calling \code{err_setstr()}, so the calling function can just return

\code{NULL}.

exception has occurred. A second variable is the `associated value'

`associated value' of the exception). Another useful function is

support so-called `callback' functions. If a C interface makes heavy

`reference-count-neutral' with respect to its arguments, but the

return value is `new': either it is a brand new object, or it is an

\chapter{Dynamic Loading}

On some systems (e.g., SunOS, SGI Irix) it is possible to configure

Python to support dynamic loading of modules implemented in C. Once

configured and installed it's trivial to use: if a Python program

executes \code{import foo}, the search for modules tries to find a

file \file{foomodule.o} in the module search path, and if one is

found, it is linked with the executing binary and executed. Once

linked, the module acts just like a built-in module.

The advantages of dynamic loading are twofold: the `core' Python

binary gets smaller, and users can extend Python with their own

modules implemented in C without having to build and maintain their

own copy of the Python interpreter. There are also disadvantages:

dynamic loading isn't available on all systems (this just means that

on some systems you have to use static loading), and dynamically

loading a module that was compiled for a different version of Python

(e.g., with a different representation of objects) may dump core.

{\bf NEW:} Under SunOS, dynamic loading now uses SunOS shared

libraries and is always configured. See at the end of this chapter

for how to create a dynamically loadable module.

\section{Configuring and building the interpreter for dynamic loading}

(Ignore this section for SunOS --- on SunOS dynamic loading is always

configured.)

Dynamic loading is a little complicated to configure, since its

implementation is extremely system dependent, and there are no

really standard libraries or interfaces for it. I'm using an

extremely simple interface, which basically needs only one function:

\begin{verbatim}

\end{verbatim}

where \code{binary} is the pathname of the currently executing program

(not just \code{argv[0]}!), \code{object} is the name of the \samp{.o}

file to be dynamically loaded, and \code{function} is the name of a

function in the module. If the dynamic loading succeeds,

\code{dl_loadmod()} returns a pointer to the named function; if not, it

returns \code{NULL}.

I provide two implementations of \code{dl_loadmod()}: one for SGI machines

running Irix 4.0 (written by my colleague Jack Jansen), and one that

is a thin interface layer for Wilson Ho's (GNU) dynamic loading

package \dfn{dld} (version 3.2.3). Dld implements a much more powerful

version of dynamic loading than needed (including unlinking), but it

does not support System V's COFF object file format. It currently

supports only VAX (Ultrix), Sun 3 (SunOS 3.4 and 4.0), SPARCstation

(SunOS 4.0), Sequent Symmetry (Dynix), and Atari ST (from the dld

3.2.3 README file). Dld is part of the standard Python distribution;

if you didn't get it,many ftp archive sites carry dld these days, so

it won't be hard to get hold of it if you need it (using archie).

(If you don't know where to get dld, try anonymous ftp to

\file{wuarchive.wustl.edu:/mirrors2/gnu/dld-3.2.3.tar.Z}. Jack's dld

can be found at \file{ftp.cwi.nl:/pub/python/dl.tar.Z}.)

To build a Python interpreter capable of dynamic loading, you need to

edit the Makefile. Basically you must uncomment the lines starting

with \samp{\#DL_}, but you must also edit some of the lines to choose

which version of dl_loadmod to use, and fill in the pathname of the dld

library if you use it. And, of course, you must first build

dl_loadmod and dld, if used. (This is now done through the Configure

script. For SunOS, everything is now automatic as long as the

architecture type is \code{sun4}.)

\section{Building a dynamically loadable module}

Building an object file usable by dynamic loading is easy, if you

follow these rules (substitute your module name for \code{foo}

everywhere):

\begin{itemize}

The source filename must be \file{foomodule.c}, so the object

name is \file{foomodule.o}.

The module must be written as a (statically linked) Python extension

module (described in an earlier chapter) except that no line for it

must be added to \file{config.c} and it mustn't be linked with the

main Python interpreter.

The module's initialization function must be called \code{initfoo}; it

must install the module in \code{sys.modules} (generally by calling

\code{initmodule()} as explained earlier.

The module must be compiled with \samp{-c}. The resulting .o file must

not be stripped.

Since the module must include many standard Python include files, it

must be compiled with a \samp{-I} option pointing to the Python source

directory (unless it resides there itself).

On SGI Irix, the compiler flag \samp{-G0} (or \samp{-G 0}) must be passed.

IF THIS IS NOT DONE THE RESULTING CODE WILL NOT WORK.

{\bf NEW:} On SunOS, you must create a shared library from your \samp{.o}

file using the following command (assuming your module is called

\code{foo}):

\begin{verbatim}

\end{verbatim}

and place the resulting \samp{.so} file in the Python search path (not

the \samp{.o} file). Note: on Solaris, you need to pass \samp{-G} to

the loader.

\end{itemize}

\section{Using libraries}

If your dynamically loadable module needs to be linked with one or

more libraries that aren't linked with Python (or if it needs a

routine that isn't used by Python from one of the libraries with which

Python is linked), you must specify a list of libraries to search

after loading the module in a file with extension \samp{.libs} (and

otherwise the same as your \samp{.o} file). This file should contain

one or more lines containing whitespace-separated absolute library

pathnames. When using the dl interface, \samp{-l...} flags may also

be used (it is in fact passed as an option list to the system linker

ld(1)), but the dl-dld interface requires absolute pathnames. I

believe it is possible to specify shared libraries here.

(On SunOS, any extra libraries must be specified on the \code{ld}

command that creates the \samp{.so} file.)

\section{Caveats}

Dynamic loading requires that \code{main}'s \code{argv[0]} contains

the pathname or at least filename of the Python interpreter.

Unfortunately, when executing a directly executable Python script (an

executable file with \samp{\#!...} on the first line), the kernel

overwrites \code{argv[0]} with the name of the script. There is no

easy way around this, so executable Python scripts cannot use

dynamically loaded modules. (You can always write a simple shell

script that calls the Python interpreter with the script as its

input.)

When using dl, the overlay is first converted into an `overlay' for

the current process by the system linker (\code{ld}). The overlay is

saved as a file with extension \samp{.ld}, either in the directory

where the \samp{.o} file lives or (if that can't be written) in a

temporary directory. An existing \samp{.ld} file resulting from a

previous run (not from a temporary directory) is used, bypassing the

(costly) linking phase, provided its version matches the \samp{.o}

file and the current binary. (See the \code{dl} man page for more

details.)