Thanks to visit codestin.com
Credit goes to github.com

Skip to content

bpo-33351: Patches to build on clang-cl #18371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Conversation

isuruf
Copy link
Contributor

@isuruf isuruf commented Feb 6, 2020

Replaces #7680

This PR also adds an azure job for testing the build with clang-cl

Also adds a distutils patch modified from the one by @zufuliu.

https://bugs.python.org/issue33351

emmatyping and others added 2 commits February 6, 2020 00:44
This was mostly:
- forward declare struct timeval in pytime (it should have been anyway)
- moving struct packing in the correct place
- Using a more portable, standard garunteed stringize macro
- defining compiler names
@isuruf
Copy link
Contributor Author

isuruf commented Feb 6, 2020

cc @zooba, @ethanhs

@zufuliu
Copy link

zufuliu commented Feb 6, 2020

compared to https://github.com/zufuliu/llvm-utils/blob/master/clang/clang-cl-py3.diff, the change to _msvccompiler.py seems outdated, specifically, PLAT_TO_LLVM_TARGETS is changed to:

PLAT_TO_LLVM_TARGETS = {
    'win32': '-m32',
    'win-amd64': '-m64',
    'win-arm64': '--target=aarch64-pc-windows-msvc',
}

@emmatyping
Copy link
Member

Neat! Thank you for pushing this forward, I've been working on the implementation for PEP 585 recently and haven't had time to complete things.

I believe the CPython CI is going to moving off of Azure pipelines and use Github actions if I am not mistaken, so perhaps using that would be better. (Though I'd wait for Steve to comment before making changes).

@isuruf
Copy link
Contributor Author

isuruf commented Feb 6, 2020

@zufuliu, I specifically used --target=x86_64-pc-windows-msvc because that's unambiguous and -m64 might mean msvc or mingw depending on the default target of the compiler.

@zufuliu
Copy link

zufuliu commented Feb 6, 2020

@isuruf OK. I don't use clang-cl from mingw-w64 LLVM (from msys2 or https://github.com/mstorsjo/llvm-mingw)

there is bug in _msvccompiler.py (it's may be better to fallback to cl.exe when clang-cl.exe isn't found):

        if self.use_clang_cl:
            self.cc = _find_exe("clang-cl.exe")
        self.cc = _find_exe("cl.exe", paths)

and I added _find_llvm in _msvccompiler.py to find LLVM from registry (added by official LLVM installers, with MSVC as default target), so don't require the LLVM\bin folder to be added to %PATH%.

Thanks to Zufu Liu (@zufuliu) for the code
@isuruf
Copy link
Contributor Author

isuruf commented Feb 6, 2020

Thanks @zufuliu, fixed it. I didn't add the registry lookup, because I usually use clang-cl from a different source and I wanted to have a way to use the one from PATH even if there's a compiler installed from official sources with the registry key. So, how about checking in PATH and then falling back to the registry? Note that some visual studio installations also have a clang.exe inside.

@codecov
Copy link

codecov bot commented Feb 6, 2020

Codecov Report

Merging #18371 into master will decrease coverage by 0.06%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #18371      +/-   ##
==========================================
- Coverage   82.19%   82.12%   -0.07%     
==========================================
  Files        1957     1955       -2     
  Lines      589224   584483    -4741     
  Branches    44428    44477      +49     
==========================================
- Hits       484334   480029    -4305     
+ Misses      95222    94807     -415     
+ Partials     9668     9647      -21     
Impacted Files Coverage Δ
Lib/distutils/tests/test_bdist_rpm.py 30.00% <0.00%> (-65.00%) ⬇️
Lib/distutils/command/bdist_rpm.py 7.63% <0.00%> (-56.88%) ⬇️
Modules/_decimal/libmpdec/umodarith.h 80.76% <0.00%> (-19.24%) ⬇️
Lib/test/test_urllib2net.py 76.92% <0.00%> (-13.85%) ⬇️
Lib/test/test_smtpnet.py 78.57% <0.00%> (-12.86%) ⬇️
Lib/ftplib.py 63.85% <0.00%> (-6.06%) ⬇️
Lib/test/test_ftplib.py 87.11% <0.00%> (-4.72%) ⬇️
Lib/dbm/__init__.py 66.66% <0.00%> (-4.45%) ⬇️
Tools/scripts/db2pickle.py 17.82% <0.00%> (-3.97%) ⬇️
Lib/test/test_socket.py 71.94% <0.00%> (-3.87%) ⬇️
... and 385 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d06953...79fe165. Read the comment docs.

@zufuliu
Copy link

zufuliu commented Feb 25, 2020

I think PLAT_TO_LLVM_TARGETS can be updated to include 32-bit ARM, at least with LLVM 9.0.1 32-bit ARM target (--target=arm-pc-windows-msvc) is also supported by clang-cl, but there are other bugs (like https://bugs.llvm.org/show_bug.cgi?id=42711) prevents the usage.

zufuliu added a commit to zufuliu/llvm-utils that referenced this pull request Feb 27, 2020
…from i386-pc-windows-msvc to i686-pc-windows-msvc.

Apply some changes by Isuru Fernando from python/cpython#18371 to clang-cl for Python3 distutils.
@zufuliu
Copy link

zufuliu commented Feb 27, 2020

There is a mistake in
target_flag = "{}-pc-windows-msvc".format(PLAT_TO_LLVM_TARGETS[plat_name])
it should be
target_flag = "--target={}-pc-windows-msvc".format(PLAT_TO_LLVM_TARGETS[plat_name])

I changed PLAT_TO_LLVM_TARGETS to

PLAT_TO_LLVM_TARGETS = {
    'win32': 'i686-pc-windows-msvc',
    'win-amd64': 'x86_64-pc-windows-msvc',
    'win-arm32' : 'arm-pc-windows-msvc',
    'win-arm64': 'aarch64-pc-windows-msvc',
}

so target flag can be written as
target_flag = '--target=' + PLAT_TO_LLVM_TARGETS[plat_name]

Also use the full target triple in the map
@isuruf
Copy link
Contributor Author

isuruf commented Feb 27, 2020

Thanks @zufuliu

@isuruf
Copy link
Contributor Author

isuruf commented Apr 9, 2020

Fixed merge conflicts. Ready for a review again.

@gongminmin
Copy link
Contributor

Any progress on this?

@bashtage
Copy link

If it is any encouragement, numpy/numpy#13816 builds NumPy and passes all tests using clang-cl.

@@ -262,6 +275,16 @@ def initialize(self, plat_name=None):
ldflags_debug = [
'/nologo', '/INCREMENTAL:NO', '/LTCG', '/DEBUG:FULL'
]
if self.use_clang_cl:
# Add target for clang
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if CPython makes use of any intrinsic or built-ins, but I had problems with both.

Intrinsics required removing some of the include directories that are used with MSVC since clang doesn't need the .h files.

Correct builtin support (on 32 bit Windows) required linking agsint the builtin .lib file.

@zufuliu
Copy link

zufuliu commented Jul 15, 2020

@isuruf It seems this PR need to be submitted to https://github.com/pypa/distutils, distutils in stdlib is going to be removed.
See pypa/setuptools#2065 and https://bugs.python.org/issue41282

zufuliu added a commit to zufuliu/llvm-utils that referenced this pull request May 9, 2021
…from i386-pc-windows-msvc to i686-pc-windows-msvc.

Apply some changes by Isuru Fernando from python/cpython#18371 to clang-cl for Python3 distutils.
@kumaraditya303
Copy link
Contributor

@isuruf Can you rebase your PR on main branch ?

@isuruf
Copy link
Contributor Author

isuruf commented Mar 22, 2022

I've already done that multiple times. I'll rebase after I get a review from someone who is interested in merging this.

@hauntsaninja
Copy link
Contributor

Thanks for the PR, but I think some of this was done in #101352 and distutils has been removed from CPython main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants