forked from neuromorphs/tonic
-
Notifications
You must be signed in to change notification settings - Fork 0
Modernise CI/CD pipeline with pyproject.toml and uv #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replace legacy pbr/setup.py build system with standards-compliant pyproject.toml using setuptools backend. This enables compatibility with modern Python packaging tools (PEP 517/518/621). Changes: - Add pyproject.toml with complete project metadata - Remove pbr dependency from requirements.txt (delete file, now redundant) - Update tonic/__init__.py to use importlib.metadata for versioning - Remove docs/requirements.txt dependency on pbr The package now builds as a universal wheel (py3-none-any) compatible with all platforms without platform-specific builds.
Introduce uv.lock to ensure reproducible builds and faster dependency installation (10-100x faster than pip). The lockfile includes all dependencies with exact versions for development, testing, and documentation.
Update CI/CD workflow to support modern Python versions and additional platforms: Platform support: - Add macOS testing (Intel and ARM64) - Maintain Ubuntu and Windows support Python version support: - Extend to Python 3.10, 3.11, 3.12, 3.13, 3.14 - Previously supported: 3.10, 3.11, 3.12 Performance improvements: - Replace pip with uv for 10-100x faster dependency installation - Add concurrency control to cancel redundant workflow runs - Enable caching for uv environments Build improvements: - Use 'uv build' instead of deprecated 'python setup.py bdist_wheel' - Update codecov action to v5 - Add permissions for OIDC token (trusted publishing ready)
Pre-commit updates: - Replace Black + isort with Ruff (10-100x faster, unified tooling) - Update to Ruff v0.14.0 (was using ancient v0.0.253 equivalent) - Add mypy v1.18.2 for type checking (informational only) - Update pre-commit-hooks to v6.0.0 - Add security hooks: detect-private-key, check-symlinks Documentation updates: - Add uv installation instructions for developers - Update platform support matrix (now includes macOS Intel/ARM64) - Document Python 3.10-3.14 support All pre-commit hooks respect configuration in pyproject.toml for consistency.
Remove obsolete pbr-based setup files now that pyproject.toml provides all package metadata and build configuration. Modern build tools (pip>=21.3, build, uv) use pyproject.toml directly via PEP 517/518.
Pin torch, torchaudio, and torchvision to 2.3.0 to match develop branch versions. This ensures the CI modernisation branch focuses purely on infrastructure changes (pyproject.toml, uv, lockfiles) without introducing dependency-related breaking changes. torchaudio 2.9+ introduces breaking changes (download_asset removal, torchcodec requirement) that should be addressed separately in a future dependency update PR.
Remove Python 3.13 and 3.14 from CI matrix due to PyTorch 2.3.0 wheel incompatibility. Add --no-build flag to prevent source builds that cause 6-hour timeouts on macOS-14. Improve Windows diagnostics for libsndfile. Changes: - Reduce matrix from 20 to 12 jobs (4 OS × 3 Python versions) - Add --no-build flag to all uv sync commands - Improve Windows PowerShell diagnostics - Document Python/PyTorch compatibility constraints PyTorch 2.3.0 lacks pre-built wheels for Python 3.13+ (requires PyTorch 2.6+) and Python 3.14+ (requires PyTorch 2.9+). The --no-build flag prevents uv from attempting multi-hour source builds when wheels are unavailable. Fixes: https://github.com/tallamjr/tonic/actions/runs/18601347188 References: pytorch/pytorch#130249 pytorch/pytorch#156856 https://docs.astral.sh/uv/guides/integration/pytorch/
The --no-build flag was preventing the local tonic package from being installed as an editable package, causing "no binary distribution" errors. Since we've already removed Python 3.13/3.14 from the matrix, and PyTorch 2.3.0 has pre-built wheels for Python 3.10-3.12 on all platforms, the --no-build flag is unnecessary. uv will automatically use wheels for all dependencies. The matrix change alone is sufficient to prevent dependency installation failures and timeouts. Fixes: error: Distribution `tonic==1.6.0 @ editable+.` can't be installed because it is marked as `--no-build` but has no binary distribution
Replace deprecated macos-13 and macos-14 runners with: - macos-15-intel: Intel (x86_64) architecture - macos-15: Apple Silicon (ARM64) architecture This provides comprehensive macOS testing on both architectures using the latest runner images, whilst avoiding the macos-13 deprecation (retiring December 4th, 2025). Matrix now tests 4 OS configurations × 3 Python versions = 12 jobs: - Ubuntu (x86_64) - Windows (x86_64) - macOS Intel (x86_64) - macOS ARM (ARM64) References: https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/ actions/runner-images#13046
PyTorch 2.3.0 lacks wheels for Intel macOS (x86_64) on macOS 15. Available wheels: Linux x86_64/ARM, macOS ARM64, Windows x86_64. Using macos-latest (ARM64) for macOS coverage only.
6 tasks
PyTorch 2.3.0 fails on Windows with NumPy 2.0+ due to missing NumPy C API (_ARRAY_API). Errors manifest as 'RuntimeError: Numpy is not available' in torch.from_numpy() calls. Constrain numpy to <2.0 to ensure compatibility across all platforms. Fixes Windows test failures with torch operations requiring numpy interop.
Configure matplotlib to use non-interactive 'Agg' backend when CI=true, preventing plt.show() from blocking on macOS where GUI frameworks are partially available. Remove Windows skip decorators as tests now work on all platforms. Fixes: https://github.com/tallamjr/tonic/actions/runs/18630080196
The imageio PNG-FI format (FreeImage plugin) is unavailable on macOS, causing test_optical_flow to fail. Implemented fallback to pypng library which provides reliable 16-bit PNG support across all platforms. References: imageio/imageio#204 https://stackoverflow.com/questions/32622658/read-16-bit-png-image-file-using-python
The documentation job was failing because sphinx-build was not in the PATH when running make commands. The virtual environment created by uv sync needs to be activated before running make, as the Makefile spawns subshells that require sphinx-build to be available. Other jobs correctly use 'uv run' to auto-activate the venv, but the documentation build was calling make directly without activation.
Removed pbr (Python Build Reasonableness) which was an obsolete build system primarily used by OpenStack projects. The version is now managed via static declaration in pyproject.toml and accessed through importlib.metadata, eliminating the need for pbr.sphinxext. Additionally, fixed Python 3.12+ SyntaxWarnings by converting docstrings containing LaTeX math notation to raw strings (r"""..."""), which properly handles backslash characters in the math expressions. Changes: - Remove pbr.sphinxext from Sphinx extensions in docs/conf.py - Remove pbr from .binder/requirements.txt - Convert docstrings to raw strings in spatial_jitter.py and transforms.py This aligns with the modernisation to uv and pyproject.toml-based packaging. References: https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
Updated astroid dependency from ~=2.15 to >=3.0,<4.0 to fix sphinx-autoapi crash when building documentation with Python 3.12. The error "'TreeRebuilder' object has no attribute 'visit_typealias'" occurred because astroid 2.15 doesn't support Python 3.12's PEP 695 type alias syntax. Astroid 3.x adds full Python 3.12 support including proper handling of the new type alias features. Version constraint allows: - Astroid 3.0+ for Python 3.12 support - Below 4.0 to avoid breaking changes - Compatible with project's Python 3.10-3.12 support range References: https://github.com/tallamjr/tonic/actions/runs/18631865540/job/53118068873 https://github.com/pylint-dev/astroid/releases https://peps.python.org/pep-0695/
Updated sphinx-autoapi from unversioned to >=3.0 to fix the error "'Module' object has no attribute 'doc'" when building documentation. Astroid 3.0 removed the deprecated 'doc' attribute in favour of 'doc_node', which broke sphinx-autoapi versions < 3.0. The fix was committed to sphinx-autoapi on July 7, 2023, and version 3.0.0+ properly supports astroid 3.x by using the new 'doc_node' attribute. This change ensures compatibility with the previously updated astroid>=3.0,<4.0 dependency. References: https://github.com/tallamjr/tonic/actions/runs/18632248069/job/53118664830 readthedocs/sphinx-autoapi#392 https://github.com/readthedocs/sphinx-autoapi/releases
Updated Sphinx from ~=5.3 to >=6.1,<8.0 and constrained sphinx-autoapi to >=3.0,<3.6 to resolve dependency conflicts. The dependency chain requires: - astroid>=3.0,<4.0 (for Python 3.12 support) - sphinx-autoapi>=3.0,<3.6 (for astroid 3.x support) - sphinx>=6.1,<8.0 (required by sphinx-autoapi 3.0-3.5) Sphinx-autoapi 3.6+ requires Sphinx >=7.4, but we use <3.6 to allow a wider Sphinx version range (6.1-7.x) for better compatibility. The previous attempt with unversioned sphinx-autoapi failed because: "sphinx-autoapi>=3.0.0,<3.6.0 depends on sphinx>=6.1.0 And because sphinx-autoapi>=3.6.0 depends on sphinx>=7.4.0 and tonic[dev] depends on sphinx>=5.3,<6.dev0, we can conclude that sphinx-autoapi>=3.0.0 and tonic[dev] are incompatible." References: https://github.com/readthedocs/sphinx-autoapi/releases
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This modernises the Tonic build system and CI/CD pipeline whilst maintaining compatibility with existing functionality and dependency versions.
Changes
pbr/setup.pyto modernpyproject.tomlwith setuptools backenduvfor fast, reproducible dependency management with lockfileStatus
Dependencies pinned to match develop branch (no breaking changes)
Infrastructure-only changes (dependency updates deferred)
See #1 for full design document and implementation details.
Closes #1