\documentclass{howto}

\usepackage{ltxmarkup}

\usepackage{times}

\usepackage{distutils}

\title{Installing Python Modules}

% The audience for this document includes people who don't know anything

% about Python and aren't about to learn the language just in order to

% install and maintain it for their users, i.e. system administrators.

% Thus, I have to be sure to explain the basics at some point:

% sys.path and PYTHONPATH at least. Should probably give pointers to

% other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc.

%

% Also, I need to take into account that most modules out there don't

% (yet) use Distutils: briefly explain the old Makefile.pre.in

% convention (maybe move material from the E&E manual to here?), and

% explain where to copy .py and .so files manually if the distribution

% doesn't provide a mechanism for doing so.

%

% Finally, it might be useful to include all the material from my "Care

% and Feeding of a Python Installation" talk in here somewhere. Yow!

\author{Greg Ward}

\authoraddress{E-mail: \email{[email protected]}}

\begin{document}

\maketitle

%\begin{abstract}

%\noindent

%Abstract this!

%\end{abstract}

\tableofcontents

\section{Introduction}

\label{sec:intro}

Although Python's extensive standard library covers many programming

needs, there often comes a time when you need to add some new

functionality to your Python installation in the form of third-party

modules. This might be necessary to support your own programming, or to

support an application that you want to use and that happens to be

written in Python.

In the past, there has been little support for adding third-party

modules to an existing Python installation. With the introduction of

the Python Distribution Utilities (Distutils for short) in Python 1.6,

this is starting to change. Not everything will change overnight,

though, so while this document concentrates on installing module

distributions that use the Distutils, we will also spend some time

dealing with the old ways.

This document is aimed primarily at the people who need to install

third-party Python modules: end-users and system administrators who just

need to get some Python application running, and existing Python

programmers who want to add some new goodies to their toolbox. You

don't need to know Python to read this document; there will be some

brief forays into using Python's interactive mode to explore your

installation, but that's it. If you're looking for information on how

to distribute your own Python modules so that others may use them, see

the ``Distributing Python Modules'' manual.

\subsection{Best case: trivial installation}

\label{sec:trivial-inst}

In the best case, someone will have prepared a special version of the

module distribution you want to install that is targeted specifically at

your platform and is installed just like any other software on your

platform. For example, the module developer might make an executable

installer available for Windows users, an RPM package for users of

RPM-based Linux systems (Red Hat, SuSE, Mandrake, and many others), a

Debian package for users of Debian-based Linux systems (Debian proper,

Caldera, Corel, etc.), and so forth.

In that case, you would download the installer appropriate to your

platform and do the usual thing with it: run it if it's an executable

installer, \code{rpm -I} it if it's an RPM, etc. You don't need to run

Python or a setup script, you don't need to compile anything---you might

not even need to read any instructions (although it's always a good idea

to do so anyways).

Of course, things will not always be that easy. You might be interested

in a module distribution that nobody has created an easy-to-use

installer for use on your platform. In that case, you'll have to start

with the source distribution released by the module's

author/maintainer. Installing from a source distribution is not too

hard, as long as the modules are packaged in the standard way. The bulk

of this document is about building and installing modules that were

packaged in the standard way.

\subsection{The new standard: Distutils}

\label{sec:new-standard}

If you download a module source distribution, you can tell pretty

quickly if was packaged and distributed in the standard way, i.e. using

the Distutils. First, the distribution's name and version number will

be featured prominently in the name of the downloaded archive, e.g.

\file{foo-1.0.tar.gz} or \file{widget-0.9.7.zip}. Next, the archive

will unpack into a similarly-named directory: \file{foo-1.0} or

\file{widget-0.9.7}. Additionally, the distribution will contain a

setup script \file{setup.py}, and a \file{README.txt} (or possibly

\file{README}), which should explain that building and installing the

module distribution is a simple matter of running

\begin{verbatim}

python setup.py install

\end{verbatim}

If all these things are true, then you already know how to build and

install the modules you've just downloaded: run the command above.

Unless you need to install things in a non-standard way or customize the

build process, you don't really need this manual. Or rather, the above

command is everything you need to get out of this manual.

\subsection{The old way: no standards}

\label{sec:old-way}

Before the Distutils, there was no infrastructure to support installing

third-party modules in a consistent, standardized way. Thus, it's not

really possible to write a general manual for installing Python modules

that don't use the Distutils; the only truly general statement that can

be made is, ``Read the module's own documentation on installation.''

However, such documentation is often woefully inadequate, assuming that

you are familiar with how the Python library is laid out and will

somehow just know where to copy various files in order for Python to

find them. Also, since there is only one way to lay out the Python

library on a given platform, this manual is a good place to learn that

layout. That way, if you do have to manually install an old,

pre-Distutils module distribution, you won't be completely on your own.

Additionally, while there has not previously been a standard

installation mechanism, Python has had some standard machinery for

building extensions on Unix since Python \XXX{version?}. This machinery

(the \file{Makefile.pre.in} file) is superseded by the Distutils, but it

will no doubt live on in older module distributions for a while. This

\file{Makefile.pre.in} mechanism is documented in the ``Extending \&

Embedding Python'' manual, but that manual is aimed at module

developers---hence, we include documentation for builders/installers

here.

All of the pre-Distutils material is tucked away in

section~\ref{sec:pre-distutils}.

\section{Standard Build and Install}

\label{sec:normal-install}

As described in section~\ref{sec:new-standard}, building and installing

a module distribution using the Distutils is usually one simple command:

\begin{verbatim}

python setup.py install

\end{verbatim}

On Unix, you'd run this command from a shell prompt; on Windows, you

have to open a command prompt window and do it there; on Mac OS ...

\XXX{what the heck do you do on Mac OS?}.

\subsection{Platform variations}

You should always run the setup command from the distribution root

directory, i.e. the top-level subdirectory that the module source

distribution unpacks into. For example, if you've just downloaded a

module source distribution \file{foo-1.0.tar.gz} onto a Unix system, the

normal thing to do is:

\begin{verbatim}

gunzip -c foo-1.0.tar.gz | tar xf - # unpacks into directory foo-1.0

cd foo-1.0

python setup.py install

\end{verbatim}

On Windows, you'd probably unpack the archive before opening the command

prompt. If you downloaded the archive file to \file{C:\bslash{}Temp}, then

it probably unpacked (depending on your software) into

\file{C:\bslash{}Temp\bslash{}foo-1.0}; from the command prompt window, you would

then run

\begin{verbatim}

cd c:\temp\foo-1.0

python setup.py install

\end{verbatim}

On Mac OS, ... \XXX{again, how do you run Python scripts on Mac OS?}

\subsection{Splitting the job up}

Running \code{setup.py install} builds and installs all modules in one

fell swoop. If you prefer to work incrementally---especially useful if

you want to customize the build process, or if things are going

wrong---you can use the setup script to do one thing at a time.

For example, you can build everything in one step, and then install

everything in a second step, by invoking the setup script twice:

\begin{verbatim}

python setup.py build

python setup.py install

\end{verbatim}

(If you do this, you will notice that running the \command{install}

command first runs the \command{build} command, which will quickly

notice that it has nothing to do, since everything in the \file{build}

directory is up-to-date.

% This will cover:

% * setup.py install (the usual thing)

% * setup.py build (if you like doing things one-at-a-time)

% * setup.py build install (not necessary unless you need to supply

% build options--ref. next section)

% * where things are installed, on Unix and Windows (Mac...?)

% * simple custom install: "install --prefix=$HOME"

\comingsoon

% takes eight args (four pairs):

% pure module distribution base + directory

% non-pure module distribution base + directory

% script base + directory

% data base + directory

% ...and will no doubt take more args in future!

\newcommand{\installscheme}[8]

{\begin{tableiii}{lll}{textrm}

{Type of file}

{Installation Directory}

{Override option}

\lineiii{pure module distribution}

{\filevar{#1}\filenq{#2}}

{\option{install-purelib}}

\lineiii{non-pure module distribution}

{\filevar{#3}\filenq{#4}}

{\option{install-platlib}}

\lineiii{scripts}

{\filevar{#5}\filenq{#6}}

{\option{install-scripts}}

\lineiii{data}

{\filevar{#7}\filenq{#8}}

{\option{install-data}}

\end{tableiii}}

\section{Alternate Installation}

\label{sec:alt-install}

Often, it is necessary or desirable to install modules to a location

other than the standard location for third-party Python modules. For

example, on a Unix system you might not have permission to write to the

standard third-party module directory. Or you might wish to try out a

module before making it a standard part of your local Python

installation; this is especially true when upgrading a distribution

already present: you want to make sure your existing base of scripts

still works with the new version before actually upgrading.

The Distutils \command{install} command is designed to make installing

module distributions to an alternate location simple and painless. The

basic idea is that you supply a base directory for the installation, and

the \command{install} command picks a set of directories (called an

\emph{installation scheme}) under this base directory in which to

install files. The details differ across platforms, so read whichever

of the following section applies to you.

\subsection{Alternate installation: Unix (the home scheme)}

\label{sec:alt-unix-prefix}

Under Unix, there are two ways to perform an alternate installation.

The ``prefix scheme'' is similar to how alternate installation works

under Windows and Mac OS, but is not necessarily the most useful way to

maintain a personal Python library. Hence, we document the more

convenient and commonly useful ``home scheme'' first.

The idea behind the ``home scheme'' is that you are building and

maintaining a personal stash of Python modules, probably under your home

directory. Installing a new module distribution is as simple as

\begin{verbatim}

python setup.py install --home # arg, doesn't work (getopt)

\end{verbatim}

or

\begin{verbatim}

python setup.py install --home=<dir>

\end{verbatim}

where you can supply any directory you like for the \option{home}

option. If you don't supply a directory (as in the first example

above), the \command{install} command uses the \code{HOME} environment

variable (or your official home directory as supplied by the password

file, if \code{HOME} is not defined).

The \option{home} option defines the installation base directory. Files

are installed to the following directories under the installation base

as follows:

\installscheme{home}{/lib/python}

{home}{/lib/python}

{home}{/bin}

{home}{/share}

\subsection{Alternate installation: Unix (the prefix scheme)}

\label{sec:alt-unix-home}

The ``prefix scheme'' is useful when you wish to use one Python

installation to perform the build/install (i.e., to run the setup

script), but install modules into the third-party module directory of a

different Python installation (or something that looks like a different

Python installation). If this sounds a trifle unusual, it is---that's

why the ``home scheme'' comes first. However, there are at least two

known cases where the prefix scheme will be useful.

First, consider that many Linux distribution put Python in \file{/usr},

rather than the more traditional \file{/usr/local}. This is entirely

appropriate, since in those cases Python is part of ``the system''

rather than a local add-on. However, if you are installing Python

modules from source, you probably want them to go in

\file{/usr/local/lib/python1.\filevar{X}} rather than

\file{/usr/lib/python1.\filevar{X}}. This can be done with

\begin{verbatim}

/usr/bin/python setup.py install --prefix=/usr/local

\end{verbatim}

Another possibility is a network filesystem where the name used to write

to a remote directory is different from the name used to read it: for

example, the Python interpreter accessed as \file{/usr/local/bin/python}

might search for modules in \file{/usr/local/lib/python1.\filevar{X}},

but those modules would have to be installed to, say,

\file{/mnt/\filevar{@server}/export/lib/python1.\filevar{X}}. This

could be done with

\begin{verbatim}

/usr/local/bin/python setup.py install --prefix=/mnt/@server/export

\end{verbatim}

In either case, the \option{prefix} option defines the installation

base, and the \option{exec-prefix} option defines the platform-specific

installation base, which is used for platform-specific files.

(Currently, this just means non-pure module distributions, but could be

expanded to C libraries, binary executables, etc.) If

\option{exec-prefix} is not supplied, it defaults to \option{prefix}.

Files are installed as follows:

\installscheme{prefix}{/lib/python1.\filevar{X}/site-packages}

{exec-prefix}{/lib/python1.\filevar{X}/site-packages}

{prefix}{/bin}

{prefix}{/share}

There is no requirement that \option{prefix} or \option{exec-prefix}

actually point to an alternate Python installation; if the directories

listed above do not already exist, they are created at installation

time.

Incidentally, the real reason the prefix scheme is important is simply

that a standard Unix installation uses the prefix scheme, but with

\option{prefix} and \option{exec-prefix} supplied by Python itself (as

\code{sys.prefix} and \code{sys.exec\_prefix}). Thus, you might think

you'll never use the prefix scheme, but every time you run \code{python

setup.py install} without any other options, you're using it.

Note that installing extensions to an alternate Python installation has

no effect on how those extensions are built: in particular, the Python

header files (\file{Python.h} and friends) installed with the Python

interpreter used to run the setup script will be used in compiling

extensions. It is your responsibility to ensure that the interpreter

used to run extensions installed in this way is compatibile with the

interpreter used to build them. The best way to ensure this is that the

two interpreters are the same version of Python (possibly different

builds, or possibly copies of the same build). (Of course, if your

\option{prefix} and \option{exec-prefix} don't even point to an

alternate Python installation, this is immaterial.)

\subsection{Alternate installation: Windows}

\label{sec:alt-windows}

Since Windows has no conception of a user's home directory, and since

the standard Python installation under Windows is simpler than that

under Unix, there's no point in having separate \option{prefix} and

\option{home} options. Just use the \option{prefix} option to specify

a base directory, e.g.

\begin{verbatim}

python setup.py install --prefix="\Temp\Python"

\end{verbatim}

to install modules to the \file{\bslash{}Temp} directory on the current

drive.

The installation base is defined by the \option{prefix} option; the

\option{exec-prefix} option is not supported under Windows. Files are

installed as follows:

\installscheme{prefix}{}

{prefix}{}

{prefix}{\bslash{}Scripts}

{prefix}{\bslash{}Data}

\subsection{Alternate installation: Mac OS}

\label{sec:alt-macos}

Like Windows, Mac OS has no notion of home directories (or even of

users), and a fairly simple standard Python installation. Thus, only a

\option{prefix} option is needed. It defines the installation base, and

files are installed under it as follows:

\XXX{how do MacPython users run the interpreter with command-line args?}

\installscheme{prefix}{:Lib}

{prefix}{:Mac:PlugIns}

{prefix}{:Scripts}

{prefix}{:Data}

\XXX{Corran Webster says: ``Modules are found in either \file{:Lib} or

\file{:Mac:Lib}, while extensions usually go in

\file{:Mac:PlugIns}''---does this mean that non-pure distributions should

be divided between \file{:Mac:PlugIns} and \file{:Mac:Lib}? If so, that

changes the granularity at which we care about modules: instead of

``modules from pure distributions'' and ``modules from non-pure

distributions'', it becomes ``modules from pure distributions'',

``Python modules from non-pure distributions'', and ``extensions from

non-pure distributions''. Is this necessary?!?}

\section{Custom Installation}

\label{sec:custom-install}

Sometimes, the alternate installation schemes described in

section~\ref{sec:alt-install} just don't do what you want. You might

want to tweak just one or two directories while keeping everything under

the same base directory, or you might want to completely redefine the

installation scheme. In either case, you're creating a \emph{custom

installation scheme}.

You probably noticed the column of ``override options'' in the tables

describing the alternate installation schemes above. Those options are

how you define a custom installation scheme. These override options can

be relative, absolute, or explicitly defined in terms of one of the

installation base directories. (There are two installation base

directories, and they are normally the same---they only differ when you

use the Unix ``prefix scheme'' and supply different \option{prefix} and

\option{exec-prefix} options.)

For example, say you're installing a module distribution to your home

directory under Unix---but you want scripts to go in

\file{\tilde/scripts} rather than \file{\tilde/bin}. As you might

expect, you can override this directory with the

\option{install-scripts} option; in this case, it makes most sense to

supply a relative path, which will be interpreted relative to the

installation base directory (your home directory, in this case):

\begin{verbatim}

python setup.py install --home --install-scripts=scripts

\end{verbatim}

Another Unix example: suppose your Python installation was built and

installed with a prefix of \file{/usr/local/python}, so under a standard

installation scripts will wind up in \file{/usr/local/python/bin}. If

you want them in \file{/usr/local/bin} instead, you would supply this

absolute directory for the \option{install-scripts} option:

\begin{verbatim}

python setup.py install --install-scripts=/usr/local/bin

\end{verbatim}

(This performs an installation using the ``prefix scheme,'' where the

prefix is whatever your Python interpreter was installed with---

\file{/usr/local/python} in this case.)

If you maintain Python on Windows, you might want third-party modules to

live in a subdirectory of \filevar{prefix}, rather than right in

\filevar{prefix} itself. This is almost as easy as customizing the

script installation directory---you just have to remember that there are

two types of modules to worry about, pure modules and non-pure modules

(i.e., modules from a non-pure distribution). For example:

\begin{verbatim}

python setup.py install --install-purelib=Site --install-platlib=Site

\end{verbatim}

The specified installation directories are relative to \filevar{prefix}.

Of course, you also have to ensure that these directories are in

Python's module search path, e.g. by putting a \file{.pth} file in

\filevar{prefix} (\XXX{should have a section describing .pth files and

cross-ref it here}).

If you want to define an entire installation scheme, you just have to

supply all of the installation directory options. The recommended way

to do this is to supply relative paths; for example, if want to maintain

all Python module-related files under \file{python} in your home

directory, and you want a separate directory for each platform that you

use your home directory from, you might define the following

installation scheme:

\begin{verbatim}

python setup.py install --home \

--install-purelib=python/lib \

--install-platlib=python/lib.$PLAT \

--install-scripts=python/scripts

--install-data=python/data

\end{verbatim}

or, equivalently,

\begin{verbatim}

python setup.py install --home=~/python \

--install-purelib=lib \

--install-platlib=lib.$PLAT \

--install-scripts=scripts

--install-data=data

\end{verbatim}

\code{\$PLAT} is not (necessarily) an environment variable---it will be

expanded by the Distutils as it parses your command line options (just

as it does when parsing your configuration file(s)).

Obviously, specifying the entire installation scheme every time you

install a new module distribution would be very tedious. Thus, you can

put these options into your Distutils config file (see

section~\ref{sec:config-files}):

\begin{verbatim}

[install]

install-base=$HOME

install-purelib=python/lib

install-platlib=python/lib.$PLAT

install-scripts=python/scripts

install-data=python/data

\end{verbatim}

or, equivalently,

\begin{verbatim}

[install]

install-base=$HOME/python

install-purelib=lib

install-platlib=lib.$PLAT

install-scripts=scripts

install-data=data

\end{verbatim}

Note that these two are \emph{not} equivalent if you supply a different

installation base directory when you run the setup script. For example,

\begin{verbatim}

python setup.py --install-base=/tmp

\end{verbatim}

would install pure modules to \filevar{/tmp/python/lib} in the first

case, and to \filevar{/tmp/lib} in the second case. (For the second

case, you probably want to supply an installation base of

\file{/tmp/python}.)

You probably noticed the use of \code{\$HOME} and \code{\$PLAT} in the

sample configuration file input. These are Distutils configuration

variables, which bear a strong resemblance to environment variables. In

fact, you can use environment variables in config files, but the

Distutils additionally define a few extra variables that may not be in

your environment, such as \code{\$PATH}. See

section~\ref{sec:config-files} for details.

\XXX{need some Windows and Mac OS examples---when would custom

installation schemes be needed on those platforms?}

\section{Distutils Configuration Files}

\label{sec:config-files}

\comingsoon

\section{Pre-Distutils Conventions}

\label{sec:pre-distutils}

\subsection{The \protect\file{Makefile.pre.in} file}

\label{sec:makefile-pre-in}

\subsection{Installing modules manually}

\label{sec:manual-install}

\end{document}