THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

greggdonovan · 2025-12-22T10:49:56Z

JIRA link for THRIFT-5915

Summary:

Switch Python setup scripts to use setuptools instead of stdlib distutils (Python 3.12 removed distutils).
Use setuptools._distutils errors for build_ext error handling when available.
Guard ntohll/htonll definitions in lib/py/src/ext/endian.h to avoid macOS macro collisions that break C++ builds.
Update tornado, twisted, and zope to the minimum versions that support Python 3.12.

Why:

Python 3.12 no longer ships distutils, causing thrift builds from sdist to fail during wheel build.
On macOS, system headers define ntohll/htonll macros; the local inline definitions collided, causing compilation failures.

Validation:

Built a wheel from lib/py with Python 3.12 successfully after the changes.

Related issues:

THRIFT-5899 - "Python tests fail for the Appveyor MSVC builds". My PR updates MSVC to 2022 and tests with Python 3.12. This is green/passing, so maybe we'll be able to close that JIRA too?

N.B.: there are multiple commits at the moment but I will squash them to a single commit before merge per guidelines.

fishy · 2025-12-22T17:22:27Z

contrib/fb303/py/setup.py

-try:
-    from setuptools import setup, Extension
-except:
-    from distutils.core import setup, Extension, Command


should we keep this (but remove Command, looks like that's unused) as fallback for earlier versions of python? (although I'm not familiar of what scenario will have no setuptools but distutils)

Thanks for taking a look, @fishy! Do you know what the Thrift support guidelines are for older python versions? e.g. We have a followup PR to add better type support, ty type checking, etc. but that's dependent on more modern python versions. The guidelines in LANGUAGES.md appear pretty outdated (i.e. python 3.6.8 is the latest supported).

Would we be open to specifying that EOL python versions are not supported (currently 3.9 and older are EOL, with 3.10 scheduled to go EOL in October 2026)? That would make it much easier to evolve type support and modernize the codebase.

In this case, I think relying on setuptools >= 61.0 and dropping distutils limits us to python 3.7+, which seems very reasonable.

there's no single support guideline for older language versions (each language has their own), but don't support EOL versions is a very reasonable and common approach.

fishy · 2025-12-22T17:22:46Z

lib/py/setup.py

-try:
-    from setuptools import setup, Extension
-except Exception:
-    from distutils.core import setup, Extension


same question here regarding fallback to earlier versions of python

lib/py/setup.py

switching to just building as a source allows us to patch in apache/thrift#3276 once that has landed/published we can switch back to pypi Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: phlax <[email protected]>

fishy · 2026-01-14T15:57:25Z

.github/workflows/build.yml

        run: make -C test/py check

+  lib-python-macos:
+    needs: compiler-macos


can this also be a matrix instead? (I'm not familiar with github actions enough so it's totally fine if this is not supported)

@fishy Thanks - let me give it a try.

fishy · 2026-01-14T15:58:39Z

.github/workflows/build.yml

+    runs-on: macos-14
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]


if needs cannot be a matrix, I wonder whether we can define this as a "variable" somewhere in this yaml, so when we change the supported python versions in the future we only need to change one place.

fishy · 2026-01-14T15:59:02Z

.github/workflows/build.yml

+
+      - name: Python setup
+        run: |
+          python -m pip install --upgrade pip setuptools wheel flake8 tornado twisted zope.interface


similar here on possibility to define the list as a variable

greggdonovan · 2026-01-15T17:20:17Z

@fishy - I think this is ready for another look. I have another PR I'm working on for THRIFT-5537 - "drop eol python versions" - but it probably makes sense to keep that separate. Thanks!

fishy

looks like everything is working now! thank you.

can you please squash the commits so I can merge it?

fishy · 2026-01-29T18:56:17Z

.github/workflows/build.yml

+    strategy:
+      matrix:
+        os: *macos_versions
+        python-version: *python_versions


- Add pyproject.toml with setuptools build requirement for PEP 517 compliance - Replace distutils imports with setuptools equivalents - Use setuptools error names directly (CompileError, ExecError, PlatformError) - Fix macOS header collision with ntohll/htonll macros in endian.h - Add a matrix of MacOS versions (macos-15-intel, macos-14, macos-15, macos-26) - Add a matrix of non-EOL Python versions for testing - Remove MSVC2015 from the test matrix (very old). - Support MSVC2022, the latest in AppVeyor. - Upgrade tornado, twisted, and zope.interface versions to the first that support Python 3.12. - Try to make the test_socket, RunClientServer, and TestServer tests less flaky. This fixes the ModuleNotFoundError: No module named 'distutils' error when building thrift with Python 3.12+.

greggdonovan · 2026-01-29T20:40:47Z

can you please squash the commits so I can merge it?

Done! Thanks for taking a look. I have a PR coming after this one to clean up the old Python versions (including old py2 code). Thanks for all the help!

greggdonovan force-pushed the etsy-python312-fix branch from b70cdb9 to ea60e94 Compare December 22, 2025 10:59

fishy reviewed Dec 22, 2025

View reviewed changes

lib/py/setup.py Outdated Show resolved Hide resolved

greggdonovan force-pushed the etsy-python312-fix branch from ea60e94 to ed73c20 Compare January 4, 2026 19:33

Jens-G added the python label Jan 7, 2026

phlax mentioned this pull request Jan 8, 2026

thrift: Dont use the pypi package envoyproxy/envoy#42908

Merged

mergeable bot added the github_actions Pull requests that update GitHub Actions code label Jan 14, 2026

fishy reviewed Jan 14, 2026

View reviewed changes

greggdonovan force-pushed the etsy-python312-fix branch from c6f9e1f to fc04dad Compare January 14, 2026 17:42

mergeable bot added the c++ label Jan 14, 2026

greggdonovan changed the title ~~Fix Python 3.12 build issues in thrift Python~~ THRIFT-5915: Fix Python 3.12 build issues in thrift Python Jan 15, 2026

greggdonovan force-pushed the etsy-python312-fix branch from 4daefe5 to 9dc0b35 Compare January 15, 2026 16:47

mergeable bot added the nodejs label Jan 15, 2026

greggdonovan marked this pull request as ready for review January 15, 2026 17:18

greggdonovan requested a review from fishy January 15, 2026 17:18

Jens-G assigned fishy Jan 22, 2026

fishy approved these changes Jan 29, 2026

View reviewed changes

.github/workflows/build.yml

strategy:

matrix:

os: *macos_versions

python-version: *python_versions

Copy link

Member

fishy Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

greggdonovan force-pushed the etsy-python312-fix branch from 7ec3fd9 to 655dafb Compare January 29, 2026 20:39

fishy approved these changes Jan 29, 2026

View reviewed changes

fishy merged commit 62ec929 into apache:master Jan 29, 2026
66 of 67 checks passed

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

Conversation

greggdonovan commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greggdonovan commented Jan 15, 2026

Uh oh!

fishy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greggdonovan commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greggdonovan commented Dec 22, 2025 •

edited

Loading