Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

@llvm/pr-subscribers-clang

Author: Ryan Mast (nightlark)

diff --git a/.git_archival.txt b/.git_archival.txt
new file mode 100644
index 000000000000000..7c5100942aae489
--- /dev/null
+++ b/.git_archival.txt
@@ -0,0 +1,3 @@
+node: $Format:%H$
+node-date: $Format:%cI$
+describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
diff --git a/.gitattributes b/.gitattributes
index 6b281f33f737db9..b94d65d60a8840a 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,3 +1,5 @@
+.git_archival.txt  export-subst
+
 libcxx/src/**/*.cpp     merge=libcxx-reformat
 libcxx/include/**/*.h   merge=libcxx-reformat
 
diff --git a/clang/bindings/python/.gitignore b/clang/bindings/python/.gitignore
new file mode 100644
index 000000000000000..da1a0b4b0aa60d2
--- /dev/null
+++ b/clang/bindings/python/.gitignore
@@ -0,0 +1,21 @@
+# setuptools_scm auto-generated version file
+_version.py
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+build/
+dist/
+*.egg-info/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
\ No newline at end of file
diff --git a/clang/bindings/python/pyproject.toml b/clang/bindings/python/pyproject.toml
new file mode 100644
index 000000000000000..1097f76f7e00787
--- /dev/null
+++ b/clang/bindings/python/pyproject.toml
@@ -0,0 +1,34 @@
+[build-system]
+requires = ["setuptools>=42", "setuptools_scm"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "clang"
+description = "libclang python bindings"
+readme = {file = "README.txt", content-type = "text/plain"}
+
+license = { text = "Apache-2.0 with LLVM exception" }
+authors = [
+    { name = "LLVM" }
+]
+keywords = ["llvm", "clang", "libclang"]
+classifiers = [
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Development Status :: 5 - Production/Stable",
+    "Topic :: Software Development :: Compilers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+]
+dynamic = ["version"]
+
+[project.urls]
+Homepage = "http://clang.llvm.org/"
+Download = "http://llvm.org/releases/download.html"
+
+[tool.setuptools_scm]
+root = "../../.."
+version_file = "clang/_version.py"
+fallback_version = "0.0.0.dev0"
+# Regex version capture group gets x.y.z with optional -rcN, -aN, -bN suffixes; -init is just consumed
+tag_regex = "^llvmorg-(?P<version>[vV]?\\d+(?:\\.\\d+)*(?:-(?:rc|a|b)\\d+)?)(?:.*)$"
\ No newline at end of file

Hey, I'm new to the process of contributing to LLVM, so let me know of any best practices or things I might have missed in the contributed guide.

I'm looking at improving the situation for packaging libclang bindings on PyPI and ideally would like maintain the things needed here alongside the python scripts that are being packaged, rather than hacking something together in a separate repository.

@Endilll the Python bindings seem like part of LLVM that you have expertise in, if you also want to review this PR.

@Endilll @LecrisUT what else do you think needs to be done here to get this approved?

@Endilll @LecrisUT what else do you think needs to be done here to get this approved?

I still would like to hear from @sighingnow. One of the reasons is that I think I've seen them attending one of the open meetings we have, so they might be around somewhere. Also I barely know anything about how to package this stuff correctly, and what the consequences would be if we get it wrong.

@Endilll - do you have way to contact them directly to tell them about this PR? I suspect they are not seeing any notifications for the mentions here, and will require asking "in-person" at a meeting. I also have a PR (opened >1 month ago) in the repository for the libclang package they maintain, and there's no indication they've seen it or my attempt to reach out to their listed email.

Are there any other people we could ask to review as an alternative, maybe @henryiii?

For reviewers, here's a summary of the potential issues that come to mind and a recap of what has been done so far to verify things are working as expected (to be clear, all of these potential issues have been addressed, and the sub-bullets are how to check that they are non-issues):

• The packaging metadata (keywords, name, author/contact info, etc) don't match what we expect
  • check if the pyproject.toml has the right classifiers, keywords, package name, etc. (open question: what email to list as the author?)
• The source distribution and wheel files built (using python -m build) have more files than necessary (these are the things that would be uploaded to PyPI and actually installed by users)
  • check if cloning the LLVM repo and then from the folder with pyproject.toml, building the sdist and wheel (created in a dist subfolder by default) works; unzip the *.whl file, and uncompress the *.tar.gz file to check if the files they have in them consist of the Python bindings and some packaging metadata files (e.g. no other LLVM source code should be present)
    • after creating a fake llvmorg-19.2.3 tag in my forked repo and doing the above, I got these two files: clang-19.2.3-py3-none-any.whl.zip(note extra layer of zipping to attach to GitHub comment) clang-19.2.3.tar.gz
  • check if the sdist and wheel files can also be pip installed: pip install clang-19.2.3.tar.gz or pip install clang-19.2.3-py3-none-any.whl
• The python bindings aren't placed in the right folder to import them when the package is pip installed
  • check if after pip installing the built wheel, Python can import the module using import clang.cindex
• The version number detected when building a wheel for tagged releases isn't correct
  • check if by tagging a fake release, and then building the sdist+wheel as mentioned above, all the file names have the right version number in them (this is key one for the CI workflow that will release files that users get)
  • check if version numbers picked up on the main branch will just be the major version number (taken from the llvmorg-N-init tag) and likely have a development version component (e.g. clang-21.post1.dev678+g42108e2298cc.tar.gz); only developers working on LLVM will ever install the package in this way and see versions like this
• Installing directly from a git URL or source archives for a release won't work
  • check if installing from a git URL works: pip install git+https://github.com/nightlark/llvm-project@add-python-packaging#subdirectory=clang/bindings/python
  • check if installing from source archive URL works: pip install https://github.com/nightlark/llvm-project/archive/refs/heads/add-python-packaging.zip#subdirectory=clang/bindings/python
    • end users of the clang Python bindings will be installing from the published PyPI package, so these methods of installation will be rarely used

In practice, the things affecting users will be issues related to the sdist and wheel files built for a specific tagged commit and the other installation methods working fit more into the category of nice-to-haves for LLVM devs.

@LecrisUT all of the situations listed work fine -- the top level bullet points are the "hypothetical issues" that someone trying to build the package could theoretically encounter, and the sub-bullet points are ways to check that the potential issue was addressed and is not actually an issue at all.

Also I barely know anything about how to package this stuff correctly, and what the consequences would be if we get it wrong.

Also regarding @Endilll. Nothing can go wrong really. This is still not connected to the CD and you would need permission from clang pypi project for which @nightlark has an issue about trolldbois/python-clang#21. The current state would be helpful to integrate early on, to allow people to experiment and integrate their CI, while any remaining nitpicks would be resolved when working on repairing the python CD (see #123937 and https://github.com/llvm/llvm-project/actions/runs/13661982395).

Also I barely know anything about how to package this stuff correctly, and what the consequences would be if we get it wrong.

Also regarding @Endilll. Nothing can go wrong really. This is still not connected to the CD and you would need permission from clang pypi project for which @nightlark has an issue about trolldbois/python-clang#21. The current state would be helpful to integrate early on, to allow people to experiment and integrate their CI, while any remaining nitpicks would be resolved when working on repairing the python CD (see #123937 and https://github.com/llvm/llvm-project/actions/runs/13661982395).

Happy to integrate with the pypi releases, trusted publishing whenever you are ready.

@nightlark Thank you for the write-up about things that can go wrong.

open question: what email to list as the author?

This is a good question I don't know answer to. Debian packages from our APT repository seems to specify LLVM Packaging Team <[email protected]> as the maintainer and their email.

Sorry for not articulating my concerns clearly enough. I'm not too concerned with bugs, as bugs will be eventually found and fixed, but I'm concerned with bugs that we'll have hard time fixing due to backwards compatibility concerns. One such thing I can imagine is users depending on a particular version string format, then we realize it doesn't match what the rest of LLVM does, and have hard time fixing that.

do you have way to contact them directly to tell them about this PR?

Apparently I mixed them up with another person who contributes to llvm-libc, so the answer is likely "no".

Are there any other people we could ask to review as an alternative, maybe @henryiii?

Sure. They are not going to have an experience with LLVM in particular, which is sad, but they will help us where we lack experience the most, i.e. Python packaging stuff.

@nightlark I guess I should make it clear that it's mostly up to me as a maintainer to find reviewers that I think this patch needs, so you shouldn't feel that you're the one blocking progress here. But I appreciate the help explaining what can go wrong, and even suggesting reviewers. Also, the fact that I'm on leave since the beginning of this week doesn't help making progress.

@nightlark (ping) are you still looking into this?

Yes, still looking into this -- got a bit busier through around mid-May though.