\section{PEP 255: Simple Generators}

In Python 2.2, generators were added as an optional feature, to be

enabled by a \code{from __future__ import generators} directive. In

2.3 generators no longer need to be specially enabled, and are now

always present; this means that \keyword{yield} is now always a

keyword. The rest of this section is a copy of the description of

generators from the ``What's New in Python 2.2'' document; if you read

it when 2.2 came out, you can skip the rest of this section.

Generators are a new feature that interacts with the iterators

introduced in Python 2.2.

You're doubtless familiar with how function calls work in Python or

C. When you call a function, it gets a private namespace where its local

variables are created. When the function reaches a \keyword{return}

statement, the local variables are destroyed and the resulting value

is returned to the caller. A later call to the same function will get

a fresh new set of local variables. But, what if the local variables

weren't thrown away on exiting a function? What if you could later

resume the function where it left off? This is what generators

provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function:

\begin{verbatim}

\end{verbatim}

A new keyword, \keyword{yield}, was introduced for generators. Any

function containing a \keyword{yield} statement is a generator

function; this is detected by Python's bytecode compiler which

compiles the function specially as a result.

When you call a generator function, it doesn't return a single value;

instead it returns a generator object that supports the iterator

protocol. On executing the \keyword{yield} statement, the generator

outputs the value of \code{i}, similar to a \keyword{return}

statement. The big difference between \keyword{yield} and a

\keyword{return} statement is that on reaching a \keyword{yield} the

generator's state of execution is suspended and local variables are

preserved. On the next call to the generator's \code{.next()} method,

the function will resume executing immediately after the

\keyword{yield} statement. (For complicated reasons, the

\keyword{yield} statement isn't allowed inside the \keyword{try} block

of a \code{try...finally} statement; read \pep{255} for a full

explanation of the interaction between \keyword{yield} and

exceptions.)

Here's a sample usage of the \function{generate_ints} generator:

\begin{verbatim}

\end{verbatim}

You could equally write \code{for i in generate_ints(5)}, or

\code{a,b,c = generate_ints(3)}.

Inside a generator function, the \keyword{return} statement can only

be used without a value, and signals the end of the procession of

values; afterwards the generator cannot return any further values.

\keyword{return} with a value, such as \code{return 5}, is a syntax

error inside a generator function. The end of the generator's results

can also be indicated by raising \exception{StopIteration} manually,

or by just letting the flow of execution fall off the bottom of the

function.

You could achieve the effect of generators manually by writing your

own class and storing all the local variables of the generator as

instance variables. For example, returning a list of integers could

be done by setting \code{self.count} to 0, and having the

\method{next()} method increment \code{self.count} and return it.

However, for a moderately complicated generator, writing a

corresponding class would be much messier.

\file{Lib/test/test_generators.py} contains a number of more

interesting examples. The simplest one implements an in-order

traversal of a tree using generators recursively.

\begin{verbatim}

\end{verbatim}

Two other examples in \file{Lib/test/test_generators.py} produce

solutions for the N-Queens problem (placing $N$ queens on an $NxN$

chess board so that no queen threatens another) and the Knight's Tour

(a route that takes a knight to every square of an $NxN$ chessboard

without visiting any square twice).

The idea of generators comes from other programming languages,

especially Icon (\url{http://www.cs.arizona.edu/icon/}), where the

idea of generators is central. In Icon, every

expression and function call behaves like a generator. One example

from ``An Overview of the Icon Programming Language'' at

\url{http://www.cs.arizona.edu/icon/docs/ipd266.htm} gives an idea of

what this looks like:

\begin{verbatim}

\end{verbatim}

In Icon the \function{find()} function returns the indexes at which the

substring ``or'' is found: 3, 23, 33. In the \keyword{if} statement,

\code{i} is first assigned a value of 3, but 3 is less than 5, so the

comparison fails, and Icon retries it with the second value of 23. 23

is greater than 5, so the comparison now succeeds, and the code prints

the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a

central concept. Generators are considered a new part of the core

Python language, but learning or using them isn't compulsory; if they

don't solve any problems that you have, feel free to ignore them.

One novel feature of Python's interface as compared to

Icon's is that a generator's state is represented as a concrete object

(the iterator) that can be passed around to other functions or stored

in a data structure.

\begin{seealso}

\seepep{255}{Simple Generators}{Written by Neil Schemenauer, Tim

Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer

and Tim Peters, with other fixes from the Python Labs crew.}

\end{seealso}

Patch \#527027: Allow building python as shared library with

--enable-shared

Memory API reworking -- which functions are deprecated?