Support compiling with clang-cl on Windows #77532
Comments
|
The clang folks have been hard at work making an ABI compatible backend to clang for Windows. Additionally they have created a cl compatible driver for clang, which can be used in lieu of cl itself. Clang-cl has been adopted to build Chrome on Windows http://blog.llvm.org/2018/03/clang-is-now-used-to-build-chrome-for.html, so I think it is stable enough to be considered for use. Clang-cl has several advantages, such as computed goto support and many other optimizations which would make Python faster on Windows. I would be happy to start contributing patches to further this goal, I already have a couple of small patches. |
|
Is this the same as the clang/llvm C1 that you can enable from inside Visual Studio? |
|
No, this is provided from llvm.org. You can find it as e.g. "Clang for Windows (64-bit)" here: http://releases.llvm.org/download.html#6.0.0 The Clang/C2 in Visual Studio is very different, and deprecated anyway. |
|
When supporting platforms comes up, there's a usual list of questions, especially for windows. I can remember two of them off the top of my head:
|
Not immediately, I don't think we should give up on the stability that currently exists with the cl based compilation. However, I think once CPython on clang-cl becomes stable, it will be compelling to switch. Clang-cl has the benefit of backwards compatibility with existing MSVC compiled c extensions, while generating a faster interpreter (perhaps 30% faster or more!).
I'm afraid not, I am just a college student :) |
|
Feel free to start creating patches so we can get an idea of what the changes would look like. Hopefully it's not that dramatic. Be very careful making performance claims without benchmarks to back it up, and ideally against multiple sets of hardware (MSVC is designed and tested to perform well across a range of processors, often by engineers who work for the manufacturer - intuition would suggest that an open source compiler is probably not 30% better all the time). Don't focus on the number right now, but do try to collect the justification before you expect or encourage others to do the work. Since the ABI is compatible, there should be no problem enabling extensions to be built using this compiler (assuming someone is willing to become a distutils maintainer, as there are currently none). You don't need to ask here to create a third-party library that enables this. I haven't heard any complaints about access to the compilers being an issue recently, so the only reasons to switch the interpreter itself would be source compatibility (essentially, the clang clib is better than our custom Win32 code) or performance. But we need a positive reason to switch support, not just the ability. |
Okay, will do. I have a few smaller patches to start with. Clang-cl tries to be as compatible as possible with cl, so I don't expect drastic changes. I'm currently trying to figure out an include issue with timeval, but so far the patches have been few and small.
I did not mean to say it would make Python 30% faster in all cases, I meant "up to 30% faster". This number is based on benchmarks of CPython with and without computed goto, and my own experiments of benchmarks comparing CPython in the WSL, and native Windows CPython releases on x86_64. But your point is well taken, and I will of course benchmark Python compiled with clang-cl once I have a complete working version.
When you say "someone to become a distutils maintainer" you mean for clang-cl specifically? If that is the case, I'm happy to add support and commit to continuing to work on clang-cl support in distutils, as I expect to use it a fair amount.
I agree there should be a good reason to move away from the MSVC compiler. The decision to move can be re-evaluated when there is a good argument to warrant it. |
|
FWIW, I would _love_ to see this. But I don't wrangle Windows myself so I can't usefully offer anything other than being happy to volunteer to run a Clang on Windows buildbot VM once there is something to actually be run. |
|
After wrangling with some missing compiler intrinsics, I've been able to get CPython to build with an almost vanilla clang-cl! I plan on upstreaming the patches to the LLVM project once I clean them up a bit. After that I will clean up the CPython patches and send a PR. I also ran performance with master built on MSVC compared to my branch on clang-cl with computed-goto enabled (I wasn't sure if there are other things that may be possible to turn on, computed goto seemed an obvious win). The results are decent, but some things, like json loads, are much slower (not sure why that is). The full report: msvc.json Performance version: 0.6.1 clang2goto.json Performance version: 0.6.1 ### 2to3 ### ### chameleon ### ### chaos ### ### crypto_pyaes ### ### deltablue ### ### django_template ### ### dulwich_log ### ### fannkuch ### ### float ### ### go ### ### hexiom ### ### html5lib ### ### json_dumps ### ### json_loads ### ### logging_format ### ### logging_silent ### ### logging_simple ### ### mako ### ### meteor_contest ### ### nbody ### ### nqueens ### ### pathlib ### ### pickle ### ### pickle_dict ### ### pickle_list ### ### pickle_pure_python ### ### pidigits ### ### python_startup ### ### python_startup_no_site ### ### raytrace ### ### regex_compile ### ### regex_dna ### ### regex_effbot ### ### regex_v8 ### ### richards ### ### scimark_fft ### ### scimark_lu ### ### scimark_monte_carlo ### ### scimark_sor ### ### scimark_sparse_mat_mult ### ### spectral_norm ### ### sqlalchemy_declarative ### ### sqlalchemy_imperative ### ### sqlite_synth ### ### sympy_expand ### ### sympy_integrate ### ### sympy_str ### ### sympy_sum ### ### telco ### ### unpack_sequence ### ### unpickle ### ### unpickle_list ### ### unpickle_pure_python ### ### xml_etree_generate ### ### xml_etree_iterparse ### ### xml_etree_parse ### ### xml_etree_process ### |
|
I sent my patches to clang-cl upstream [1]. It seems they want to implement Hardware Lock Elision (which is used by some MSVC compiler intrinsics in pyatomic.h) before implementing the needed intrinsics. I have found temporary replacements that do not elide locks, but have effectively the same functional purpose as those intrinsics, so I should have a full PR for CPython ready soon. |
|
I just updated the PR with some more information after trying this on every VS project. It seems that clang-cl still fails on some projects/tests, but I don't think that is a big problem. I was mostly interested in getting Python core to build with clang-cl, which it does (and passes all test with it). I will keep iterating on this as time allows. I also think it would be helpful to have an idea of the expectation for review/merge. |
|
I will add to this issue my only compile problem I had using clang-cl ver 9. It has an issue with parsing the Modules/_tracemalloc.c(64,11): error: declaration of anonymous struct must be a definition I commented on and suggested a fix for _tracemalloc.c here: |
|
With below patch, I can compile python 3.10.5 with clang 15. |
…L on Windows (GH-101352) To use this, ensure that clang support was selected in Visual Studio Installer, then set the PlatformToolset environment variable to "ClangCL" and build as normal from the command line. It remains unsupported, but at least is possible now for experimentation.
|
I wanted this, so I applied my own patch. It also suppresses a lot of uninteresting warnings, fixes a few changes since the last patch was posted here, and enables LTO on release builds. No doubt we'll have another one of these issues in the future, but for now, I'll consider this closed. |
…ClangCL on Windows (pythonGH-101352) To use this, ensure that clang support was selected in Visual Studio Installer, then set the PlatformToolset environment variable to "ClangCL" and build as normal from the command line. It remains unsupported, but at least is possible now for experimentation.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: