-
-
Notifications
You must be signed in to change notification settings - Fork 34.5k
gh-146636: PEP 803: Reference documentation #148013
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
8e203eb
c1d7fef
f83f6b4
b3e31c6
2372da9
b3799de
3bc47ee
30d97b1
e7e102a
fec46db
47ff800
f279be8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -51,144 +51,204 @@ It is generally intended for specialized, low-level tools like debuggers. | |||||||
| Projects that use this API are expected to follow | ||||||||
| CPython development and spend extra effort adjusting to changes. | ||||||||
|
|
||||||||
| .. _stable-abi: | ||||||||
| .. _stable-application-binary-interface: | ||||||||
|
|
||||||||
| Stable Application Binary Interface | ||||||||
| =================================== | ||||||||
| Stable Application Binary Interfaces | ||||||||
| ==================================== | ||||||||
|
|
||||||||
| For simplicity, this document talks about *extensions*, but the Limited API | ||||||||
| and Stable ABI work the same way for all uses of the API β for example, | ||||||||
| embedding Python. | ||||||||
| Python's :dfn:`Stable ABI` allows extensions to be compatible with multiple | ||||||||
| versions of Python, without recompilation. | ||||||||
|
|
||||||||
| .. _limited-c-api: | ||||||||
| .. note:: | ||||||||
|
|
||||||||
| Limited C API | ||||||||
| ------------- | ||||||||
| For simplicity, this document talks about *extensions*, but the Stable ABI | ||||||||
| works the same way for all uses of the API β for example, embedding Python. | ||||||||
|
|
||||||||
| Python 3.2 introduced the *Limited API*, a subset of Python's C API. | ||||||||
| Extensions that only use the Limited API can be | ||||||||
| compiled once and be loaded on multiple versions of Python. | ||||||||
| Contents of the Limited API are :ref:`listed below <limited-api-list>`. | ||||||||
| A Stable ABI is *versioned* using the first two numbers of the Python version. | ||||||||
| For example, Stable ABI 3.14 corresponds to Python 3.14. | ||||||||
| An extension compiled for Stable ABI 3.x is ABI-compatible with Python 3.x | ||||||||
| and above. | ||||||||
|
|
||||||||
| .. c:macro:: Py_LIMITED_API | ||||||||
| There are two Stable ABIs: | ||||||||
|
|
||||||||
| Define this macro before including ``Python.h`` to opt in to only use | ||||||||
| the Limited API, and to select the Limited API version. | ||||||||
| - ``abi3``, introduced in Pyton 3.2, is compatible with | ||||||||
| non-:term:`free threaded <free-threaded build>` builds of CPython. | ||||||||
|
|
||||||||
| Define ``Py_LIMITED_API`` to the value of :c:macro:`PY_VERSION_HEX` | ||||||||
| corresponding to the lowest Python version your extension supports. | ||||||||
| The extension will be ABI-compatible with all Python 3 releases | ||||||||
| from the specified one onward, and can use Limited API introduced up to that | ||||||||
| version. | ||||||||
| - ``abi3t``, introduced in Pyton 3.15, is compatible with | ||||||||
|
encukou marked this conversation as resolved.
Outdated
|
||||||||
| :term:`free threaded <free-threaded build>` builds of CPython. | ||||||||
| It has stricter API limitations than ``abi3``. | ||||||||
|
|
||||||||
| Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum | ||||||||
| minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when | ||||||||
| compiling with future Python versions. | ||||||||
| .. versionadded:: next | ||||||||
|
|
||||||||
| You can also define ``Py_LIMITED_API`` to ``3``. This works the same as | ||||||||
| ``0x03020000`` (Python 3.2, the version that introduced Limited API). | ||||||||
| ``abi3t`` was added in :pep:`803` | ||||||||
|
|
||||||||
| .. c:macro:: Py_TARGET_ABI3T | ||||||||
| It is possible for an extension to be compiled for *both* ``abi3`` and | ||||||||
| ``abi3t`` at the same time. | ||||||||
| Currently, this has no downsides compared to compiling for ``abi3t`` only. | ||||||||
|
|
||||||||
| Define this macro before including ``Python.h`` to opt in to only use | ||||||||
| the Limited API for :term:`free-threaded builds <free-threaded build>`, | ||||||||
| and to select the Limited API version. | ||||||||
| The Stable ABIs come with several caveats: | ||||||||
|
|
||||||||
| .. seealso:: :pep:`803` | ||||||||
| - Extensions that target a stable ABI must only use a limited subset of | ||||||||
| the C API. This subset is known as the :dfn:`Limited API`; its contents | ||||||||
| are :ref:`listed below <limited-api-list>`. | ||||||||
|
|
||||||||
| .. versionadded:: next | ||||||||
| - Compiling for a Stable ABI will disable some optimizations. | ||||||||
| In particular, common functions cannot be inlined to take advantage of the | ||||||||
| internal implementation details. | ||||||||
|
|
||||||||
| - Stable ABI prevents *ABI* issues, like linker errors due to missing | ||||||||
| symbols or data corruption due to changes in structure layouts or function | ||||||||
| signatures. | ||||||||
| However, other changes in Python can change the *behavior* of extensions. | ||||||||
| See Python's Backwards Compatibility Policy (:pep:`387`) for details. | ||||||||
|
|
||||||||
| .. _stable-abi: | ||||||||
| On Windows, extensions that use a Stable ABI should be linked against | ||||||||
| ``python3.dll`` rather than a version-specific library such as | ||||||||
| ``python39.dll``. | ||||||||
| This library only exposes the relevant symbols. | ||||||||
|
|
||||||||
| On some platforms, Python will look for and load shared library files named | ||||||||
| with the ``abi3`` or ``abi3t`` tag (e.g. ``mymodule.abi3.so``). | ||||||||
| :term:`Free threaded <free-threaded build>` interpreters only recognize the | ||||||||
| ``abi3t`` tag, while non-free threaded ones will prefer ``abi3`` but fall back | ||||||||
|
hugovk marked this conversation as resolved.
Outdated
|
||||||||
| to ``abi3t``. | ||||||||
| Thus, extensions compatible with both flavors should use the ``abi3t`` tag. | ||||||||
|
|
||||||||
| Python does not check if such extensions conform to a Stable ABI. | ||||||||
| Extension authors are encouraged to check using the :c:macro:`Py_mod_abi` | ||||||||
| slot or the :c:func:`PyABIInfo_Check` function, but, the user | ||||||||
| (or their packaging tool) is ultimately responsible for ensuring that, | ||||||||
| for example, extensions built for Stable ABI 3.10 are not installed for lower | ||||||||
| versions of Python. | ||||||||
|
|
||||||||
| All functions in the Stable ABI are present as functions in Python's shared | ||||||||
| library, not solely as macros. | ||||||||
| They are usable from languages that don't use the C preprocessor, | ||||||||
| such as Python via :py:mod:`ctypes`. | ||||||||
|
|
||||||||
| Stable ABI | ||||||||
| ---------- | ||||||||
|
|
||||||||
| To enable this, Python provides a *Stable ABI*: a set of symbols that will | ||||||||
| remain ABI-compatible across Python 3.x versions. | ||||||||
| Compiling for Stable ABI | ||||||||
| ------------------------ | ||||||||
|
|
||||||||
| .. note:: | ||||||||
|
|
||||||||
| The Stable ABI prevents ABI issues, like linker errors due to missing | ||||||||
| symbols or data corruption due to changes in structure layouts or function | ||||||||
| signatures. | ||||||||
| However, other changes in Python can change the *behavior* of extensions. | ||||||||
| See Python's Backwards Compatibility Policy (:pep:`387`) for details. | ||||||||
| When using a build tool (for example, ``setuptools``), the tool is | ||||||||
|
encukou marked this conversation as resolved.
Outdated
|
||||||||
| generally responsible for setting macros and synchronizing them with | ||||||||
| extension filenames and other metadata. | ||||||||
| Prefer using the tool's options over defining the macros manually. | ||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe setuptools is a poor example to use above then? Because setuptools expects users to set the build flags manually... You could delete the "(For example, SetupTools)" above, or maybe refer to meson-python or scikit-build-core instead? These days that's what I recommend to people depending on if they already use CMake or not.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AFAIK Setuptools is the most widely known build tool; the example is there to clarify what kind of βbuild toolβ I'm talking about. Setuptools has a |
||||||||
|
|
||||||||
| The Stable ABI contains symbols exposed in the :ref:`Limited API | ||||||||
| <limited-c-api>`, but also other ones β for example, functions necessary to | ||||||||
| support older versions of the Limited API. | ||||||||
| The rest of this section is relevant for tool authors, and for people who | ||||||||
| compile extensions manually. | ||||||||
|
|
||||||||
| On Windows, extensions that use the Stable ABI should be linked against | ||||||||
| ``python3.dll`` rather than a version-specific library such as | ||||||||
| ``python39.dll``. | ||||||||
| .. seealso:: `list of recommended tools`_ in the Python Packaging User Guide | ||||||||
|
|
||||||||
| On some platforms, Python will look for and load shared library files named | ||||||||
| with the ``abi3`` tag (e.g. ``mymodule.abi3.so``). | ||||||||
| It does not check if such extensions conform to a Stable ABI. | ||||||||
| The user (or their packaging tools) need to ensure that, for example, | ||||||||
| extensions built with the 3.10+ Limited API are not installed for lower | ||||||||
| versions of Python. | ||||||||
| .. _list of recommended tools: https://packaging.python.org/en/latest/guides/tool-recommendations/#build-backends-for-extension-modules | ||||||||
|
|
||||||||
| All functions in the Stable ABI are present as functions in Python's shared | ||||||||
| library, not solely as macros. This makes them usable from languages that don't | ||||||||
| use the C preprocessor. | ||||||||
| To compile for a Stable ABI, define one or both of the following macros | ||||||||
| before including ``Python.h`` to the lowest Python version your extension | ||||||||
| should support, in :c:macro:`Py_PACK_VERSION` format. | ||||||||
| Typically, you should choose a specific value rather than the version of | ||||||||
| the Python headers you are compiling against. | ||||||||
|
|
||||||||
| Since the :c:macro:`Py_PACK_VERSION` is not available before including | ||||||||
| ``Python.h``, you will need to use the number directly. | ||||||||
| For reference, the values for a few Python versions are: | ||||||||
|
|
||||||||
| Limited API Scope and Performance | ||||||||
| --------------------------------- | ||||||||
| .. version-hex-cheatsheet:: | ||||||||
|
|
||||||||
| The goal for the Limited API is to allow everything that is possible with the | ||||||||
| When the macro(s) are defined, ``Python.h`` will only expose API that is | ||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||
| compatible with the given Stable ABI -- that is, the | ||||||||
| :ref:`Limited API <limited-api-list>` plus some definitions that need to be | ||||||||
| visible to the compiler but should not be used directly. | ||||||||
|
|
||||||||
| .. c:macro:: Py_LIMITED_API | ||||||||
|
|
||||||||
| Target ``abi3``, that is, | ||||||||
| non-:term:`free threaded <free-threaded build>` builds of CPython. | ||||||||
|
|
||||||||
| .. c:macro:: Py_TARGET_ABI3T | ||||||||
|
|
||||||||
| Target ``abi3t``, that is, | ||||||||
| :term:`free threaded <free-threaded build>` builds of CPython. | ||||||||
|
|
||||||||
| .. versionadded:: next | ||||||||
|
|
||||||||
| Despite the different naming, the macros are similar; | ||||||||
| the name :c:macro:`!Py_LIMITED_API` is kept for backwards compatibility. | ||||||||
|
|
||||||||
| .. admonition:: Historical note | ||||||||
|
|
||||||||
| You can also define ``Py_LIMITED_API`` as ``3``. This works the same as | ||||||||
| ``0x03020000`` (Python 3.2, the version that introduced Stable ABI). | ||||||||
|
|
||||||||
| When both are defined, ``Python.h`` may, or may not, redefine | ||||||||
| :c:macro:`!Py_LIMITED_API` to match :c:macro:`!Py_TARGET_ABI3T`. | ||||||||
|
|
||||||||
| On a a :term:`free-threaded build` -- that is, when | ||||||||
|
encukou marked this conversation as resolved.
Outdated
|
||||||||
| :c:macro:`Py_GIL_DISABLED` is defined -- :c:macro:`!Py_TARGET_ABI3T` | ||||||||
| defaults to the value of :c:macro:`!Py_TARGET_ABI3T`. | ||||||||
|
encukou marked this conversation as resolved.
Outdated
|
||||||||
| This means that there are two ways to build for both ``abi3`` and ``abi3t``: | ||||||||
|
|
||||||||
| - define both :c:macro:`!Py_LIMITED_API` and :c:macro:`!Py_TARGET_ABI3T`, or | ||||||||
| - define only :c:macro:`!Py_LIMITED_API` and build for free-threaded Python. | ||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think defining Py_GIL_DISABLED "manually" is still only supported on Windows. |
||||||||
|
|
||||||||
|
|
||||||||
| Stable ABI Scope and Performance | ||||||||
| -------------------------------- | ||||||||
|
|
||||||||
| The goal for Stable ABI is to allow everything that is possible with the | ||||||||
| full C API, but possibly with a performance penalty. | ||||||||
| Generally, compatibility with Stable ABI will require some changes to an | ||||||||
| extension's source code. | ||||||||
|
|
||||||||
| For example, while :c:func:`PyList_GetItem` is available, its βunsafeβ macro | ||||||||
| For example, while :c:func:`PyList_GetItem` is available, its "unsafe" macro | ||||||||
| variant :c:func:`PyList_GET_ITEM` is not. | ||||||||
| The macro can be faster because it can rely on version-specific implementation | ||||||||
| details of the list object. | ||||||||
|
|
||||||||
| Without ``Py_LIMITED_API`` defined, some C API functions are inlined or | ||||||||
| replaced by macros. | ||||||||
| Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as | ||||||||
| For another example, when *not* compiling for Stable ABI, some C API | ||||||||
| functions are inlined or replaced by macros. | ||||||||
| Compiling for Stable ABI disables this inlining, allowing stability as | ||||||||
| Python's data structures are improved, but possibly reducing performance. | ||||||||
|
|
||||||||
| By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile | ||||||||
| a Limited API extension with a version-specific ABI. This can improve | ||||||||
| performance for that Python version, but will limit compatibility. | ||||||||
| Compiling with ``Py_LIMITED_API`` will then yield an extension that can be | ||||||||
| distributed where a version-specific one is not available β for example, | ||||||||
| for prereleases of an upcoming Python version. | ||||||||
| By leaving out the :c:macro:`!Py_LIMITED_API`` or :c:macro:`!Py_TARGET_ABI3T` | ||||||||
| definition, it is possible to compile Stable-ABI-compatible source | ||||||||
| for a version-specific ABI, possibly improving performance for a specific | ||||||||
| Python version. | ||||||||
|
|
||||||||
|
|
||||||||
| Limited API Caveats | ||||||||
| ------------------- | ||||||||
|
|
||||||||
| Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that | ||||||||
| code conforms to the :ref:`Limited API <limited-c-api>` or the :ref:`Stable ABI | ||||||||
| <stable-abi>`. ``Py_LIMITED_API`` only covers definitions, but an API also | ||||||||
| includes other issues, such as expected semantics. | ||||||||
| Note that compiling with :c:macro:`Py_LIMITED_API` or :c:macro:`Py_TARGET_ABI3T` | ||||||||
| is *not* a complete guarantee that code will be compatible with the | ||||||||
| expected Python versions. | ||||||||
| The macros only cover definitions, not other issues such as expected semantics. | ||||||||
|
|
||||||||
| One issue that ``Py_LIMITED_API`` does not guard against is calling a function | ||||||||
| One issue that the macros do not guard against is calling a function | ||||||||
| with arguments that are invalid in a lower Python version. | ||||||||
| For example, consider a function that starts accepting ``NULL`` for an | ||||||||
| argument. In Python 3.9, ``NULL`` now selects a default behavior, but in | ||||||||
| Python 3.8, the argument will be used directly, causing a ``NULL`` dereference | ||||||||
| and crash. A similar argument works for fields of structs. | ||||||||
|
|
||||||||
| Another issue is that some struct fields are currently not hidden when | ||||||||
| ``Py_LIMITED_API`` is defined, even though they're part of the Limited API. | ||||||||
| the macros are defined, even though they're part of the Limited API. | ||||||||
|
|
||||||||
| For these reasons, we recommend testing an extension with *all* minor Python | ||||||||
| versions it supports, and preferably to build with the *lowest* such version. | ||||||||
| versions it supports. | ||||||||
|
|
||||||||
| We also recommend reviewing documentation of all used API to check | ||||||||
| if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API`` | ||||||||
| defined, a few private declarations are exposed for technical reasons (or | ||||||||
| even unintentionally, as bugs). | ||||||||
|
|
||||||||
| Also note that the Limited API is not necessarily stable: compiling with | ||||||||
| ``Py_LIMITED_API`` with Python 3.8 means that the extension will | ||||||||
| run with Python 3.12, but it will not necessarily *compile* with Python 3.12. | ||||||||
| In particular, parts of the Limited API may be deprecated and removed, | ||||||||
| Also note that while compiling with ``Py_LIMITED_API`` 3.8 means that the | ||||||||
| extension will *load* on Python 3.12, and *compile* with Python 3.12, | ||||||||
| the same source will not necessarily compile with ``Py_LIMITED_API`` | ||||||||
| set to 3.12. | ||||||||
| In general: parts of the Limited API may be deprecated and removed, | ||||||||
| provided that the Stable ABI stays stable. | ||||||||
|
|
||||||||
|
|
||||||||
|
|
@@ -199,12 +259,12 @@ Platform Considerations | |||||||
|
|
||||||||
| ABI stability depends not only on Python, but also on the compiler used, | ||||||||
| lower-level libraries and compiler options. For the purposes of | ||||||||
| the :ref:`Stable ABI <stable-abi>`, these details define a βplatformβ. They | ||||||||
| the :ref:`Stable ABIs <stable-abi>`, these details define a βplatformβ. They | ||||||||
| usually depend on the OS type and processor architecture | ||||||||
|
|
||||||||
| It is the responsibility of each particular distributor of Python | ||||||||
| to ensure that all Python versions on a particular platform are built | ||||||||
| in a way that does not break the Stable ABI. | ||||||||
| in a way that does not break the Stable ABIs, or the version-specific ABIs. | ||||||||
| This is the case with Windows and macOS releases from ``python.org`` and many | ||||||||
| third-party distributors. | ||||||||
|
|
||||||||
|
|
@@ -365,12 +425,13 @@ The full API is described below for advanced use cases. | |||||||
| .. versionadded:: 3.15 | ||||||||
|
|
||||||||
|
|
||||||||
| .. _limited-c-api: | ||||||||
| .. _limited-api-list: | ||||||||
|
|
||||||||
| Contents of Limited API | ||||||||
| ======================= | ||||||||
|
|
||||||||
|
|
||||||||
| Currently, the :ref:`Limited API <limited-c-api>` includes the following items: | ||||||||
| This is the definitive list of :ref:`Limited API <limited-c-api>` for | ||||||||
| Python |version|: | ||||||||
|
|
||||||||
| .. limited-api-list:: | ||||||||
Uh oh!
There was an error while loading. Please reload this page.