Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@Tobias314
Copy link
Contributor

@Tobias314 Tobias314 commented Sep 3, 2025

Adds an additional job run to the build-pycolmap.yml GitHub action to build a Linux x86-64 Python wheel with CUDA support named pycolmap_cuda

This is achieved by the following changes:

  • use CUDA-enabled sameli/manylinux_2_34_x86_64_cuda_12.8 container image for cibuildwheel
  • modify repair wheel command to remove shared CUDA libraries from wheel
  • dynamically modify the pyproject.toml file during the CUDA build to name the resulting package pycolmap_cuda to avoid name collision with the non-CUDA pycolmap package

TODO:

  • decide on name of CUDA pycolmap package for publishing on PyPI -> pycolmap-cuda-12 is used

Tobias314 added 25 commits July 7, 2025 23:49
… github actions workflow for pycolmap CUDA Linux built, running workflows only when triggered
@Tobias314 Tobias314 marked this pull request as ready for review September 3, 2025 22:09
@sarlinpe
Copy link
Member

sarlinpe commented Sep 3, 2025

Thank you for your contribution. IIUC you suggest having a different package name for CUDA wheels. How would you then handle different versions of CUDA? PyTorch only uploads to PyPI a single CUDA version (12.8 currently) and hosts externally other versions and their dependencies (example: https://download.pytorch.org/whl/cu129). NVIDIA does have some CUDA shared libraries exposed as PyPI packages, are they useful for us?

@Tobias314
Copy link
Contributor Author

How would you then handle different versions of CUDA?

That is not really tackled so far.
Currently, everything is built with CUDA version 12.8 (using the sameli/manylinux_2_34_x86_64_cuda_12.8 container image).
Without further adjustments, cibuildwheel would use repairwheel to include required shared libraries into the wheel. Upon inspecting such a wheel I found this to include corresponding versions of libcudart and libcurand.
Since including them in the wheel comes with several problems (e.g. wheel size exceeding PyPI limits), I modified the repairwheel command to not include those CUDA libraries. So currently, getting the right versions of those dependencies as well as a working CUDA runtime would probably be left to the user. That is not ideal.

One way to improve this might be to dynamically modify the pyproject.toml file while building the CUDA wheel to add CUDA dependencies via PyPI packages. There seem to exist cuda-runtime and curand packages on PyPI which include those shared libraries. The PyTorch wheel also seems to have them as dependencies.

Offering multiple CUDA versions would probably go even further than this. Maybe one option would be to build different wheels using container images with different CUDA versions and different versions of those Nvidia PyPI packages specified as dependencies in the pyproject.toml? To differentiate the wheels on PyPI one could include the CUDA version in the package name (e.g. pycolmap_cuda_12.8).

PyTorch also seems to build multiple wheels for multiple CUDA versions. However, they do not differentiate them by name. Instead, it seems like they are hosting packages for different CUDA versions in different package indices. However, this probably requires self-hosting those alternative package indices.

I think an underlying problem is that wheel tags include only operating system and CPU architecture information. So dealing with things like different CUDA versions might always require getting a bit creative.

However, I do not have much experience with building CUDA-enabled Python packages. So I would be more than happy to learn about some alternative ideas.

@Tobias314
Copy link
Contributor Author

Would it make sense to update pyproject.toml file using Python instead of sed

Sure, I have basically added the Python script that you suggested above.

Copy link
Member

@sarlinpe sarlinpe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! The CUDA build increases the size of the pycolmap .so by 2x (54MB to 110MB), I guess that this is expected?

@sarlinpe sarlinpe requested a review from ahojnnes October 14, 2025 10:24
@Tobias314
Copy link
Contributor Author

The CUDA build increases the size of the pycolmap .so by 2x (54MB to 110MB), I guess that this is expected?

I guess an increase in size is expected. However, for me it looks like the .whl is only 62MB in size. I think this can be seen in the summary at https://github.com/colmap/colmap/actions/runs/18450241678/attempts/2#summary-52636181721.
Where did you get the 110MB from?

Also, I think PyPI has an upper limit of 100MB by default (https://docs.pypi.org/project-management/storage-limits/). So publishing something larger than this limit might require special permission from PyPI.

@sarlinpe
Copy link
Member

I was referring to the .so file (obtain by unpacking the wheel, which is just a zip file), the wheel size is indeed fine for PyPI (3x larger though).

@sarlinpe
Copy link
Member

sarlinpe commented Nov 3, 2025

@ahojnnes @B1ueber2y any opinion against merging this?

Copy link
Contributor

@B1ueber2y B1ueber2y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Very nice feature.

@sarlinpe sarlinpe enabled auto-merge (squash) November 3, 2025 11:40
@sarlinpe sarlinpe merged commit df73a0c into colmap:main Nov 3, 2025
13 checks passed
@sarlinpe
Copy link
Member

sarlinpe commented Nov 7, 2025

@Tobias314 Thank you for your work, we now have pycolmap-cuda12 on PyPI: https://pypi.org/project/pycolmap-cuda12/ Would you mind removing https://pypi.org/project/pycolmap-cuda/ (I believe that it's owned by you) to prevent any confusion? thank you!

@Tobias314
Copy link
Contributor Author

Tobias314 commented Nov 7, 2025

we now have pycolmap-cuda12 on PyPI: https://pypi.org/project/pycolmap-cuda12/ Would you mind removing https://pypi.org/project/pycolmap-cuda/ (I believe that it's owned by you) to prevent any confusion? thank you!

Great to hear!
I have archived pycolmap-cuda so that it can no longer be found via PyPI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants