Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG, DIST: fix normalize IBMZ features flags #20588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 15, 2021
Merged

BUG, DIST: fix normalize IBMZ features flags #20588

merged 1 commit into from
Dec 15, 2021

Conversation

seiko2plus
Copy link
Member

@seiko2plus seiko2plus commented Dec 15, 2021 •
edited
Loading

The normal behavior is to erase any flag that starts with
-m[a-z0-9\-\.] when flag -march or -mcpu specified. for example:

cc_normalize_flags([
   '-msse', '-msse2', '-msse3', '-mssse3', '-march=core-avx2'
])

should be normalized to:

['-march=core-avx2']

but in the case of s390x, on GCC flag -march=arch[0-9] doesn't implies flag
-mzvector which is required to enable zvector api. for example:

cc_normalize_flags([
   '-mzvector', '-march=arch11', '-march=arch12', '-march=arch13']
)

should be normalized to:

['-mzvector', '-march=arch13']

instead of:

['-march=arch13']

The test unit didn't detect this bug because of a single-letter typo(my bad).

Build error:

WARN: CCompilerOpt.dist_test[615] : CCompilerOpt._dist_test_spawn[749] : Command (gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Werror=vla -Werror=nonnull -Werror=pointer-arith -Werror=implicit-function-declaration -Wlogical-op -Wno-sign-compare -Werror=undef -fPIC -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/travis/build/numpy/numpy/builds/venv/include -I/opt/python/3.8.12/include/python3.8 -Ibuild/src.linux-s390x-3.8/numpy/core/src/common -Ibuild/src.linux-s390x-3.8/numpy/core/src/npymath -c /home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.c -o /tmp/tmpa6x2ndqc/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.o -MMD -MF /tmp/tmpa6x2ndqc/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.o.d -march=arch12 -Werror) failed with exit status 1 output -> 
/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.c:1:6: error: "__VEC__" is not defined, evaluates to 0 [-Werror=undef]
    1 | #if (__VEC__ < 10302) || (__ARCH__ < 12)
      |      ^~~~~~~
/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.c:2:6: error: #error VXE not supported
    2 |     #error VXE not supported
      |      ^~~~~
/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.c: In function ‘main’:
/home/travis/build/numpy/numpy/numpy/distutils/checks/cpu_vxe.c:8:5: error: ‘__vector’ undeclared (first use in this function); did you mean ‘vec_or’?
    8 |     __vector float x = vec_nabs(vec_xl(argc, (float*)argv));
      |     ^~~~~~~~
      |     vec_or

relates #20552

@seiko2plus seiko2plus added 00 - Bug component: numpy.distutils component: SIMD Issues in SIMD (fast instruction sets) code or machinery 36 - Build Build related PR labels Dec 15, 2021
@seiko2plus seiko2plus mentioned this pull request Dec 15, 2021
Merged
@seiko2plus
BUG, DIST: fix normalize IBMZ features flags
3e81230 
  The normal behavior is to erase any flag that starts with
  `-m[a-z0-9\-\.]` when flag `-march` or `-mcpu` specified. for example:

    cc_normalize_flags([
       '-msse', '-msse2', '-msse3', '-mssse3', '-march=core-avx2'
    ])

    should be normalized to: ['-march=core-avx2']

  but in the case of `s390x`, on GCC flag `-march=arch[0-9]` doesn't  implies flag
  `-mzvector` which is required to enable zvector api. for example:

    cc_normalize_flags([
       '-mzvector', '-march=arch11', '-march=arch12', '-march=arch13']
    )

  should be normalized to: ['-mzvector', '-march=arch13']

  instead of: ['-march=arch13']
@seiko2plus
Copy link
Member Author

reword the commit

@seiko2plus seiko2plus force-pushed the ccopt_test_implies_flags branch from 08a27ad to 3e81230 Compare December 15, 2021 14:46
@mattip
Copy link
Member

mattip commented Dec 15, 2021

While this does seem like the correct fix for the problem, this whole dance seems fragile. How did we get into the situation where we need to remove compiler flags: who is adding them in the first place? Perhaps it would be better to raise an error if we detect wrong flags rather than trying to fix them?

@seiko2plus
Copy link
Member Author

this whole dance seems fragile

A natural reaction to the chaos of compilers, the stage is stable compilers don't usually change their cli options.

How did we get into the situation where we need to remove compiler flags: who is adding them in the first place?

The answer is the implied features. This issue occurred when more than one feature implies one another and each one of them has a flag override the other or conflict with another or requires arch concatenation + in case of the arm.

Perhaps it would be better to raise an error if we detect wrong flags rather than trying to fix them?

there are no wrong flags, as I said above issue occurred when we combine the flags of multiple features. distutils/tests/test_ccompiler_opt.py covers all these cases.

@mattip mattip merged commit 06e37a3 into numpy:main Dec 15, 2021
@mattip
Copy link
Member

mattip commented Dec 15, 2021

Thanks @seiko2plus

@mattip
Copy link
Member

mattip commented Dec 15, 2021

The answer is the implied features. This issue occurred when more than one feature implies one another and each one of them has a flag override the other or conflict with another or requires arch concatenation

Could we hold some kind of state, and only turn them into compiler flags when all the detection is finished? Then we could manage the state in a more straight-forward fashion.

@seiko2plus
Copy link
Member Author

Could we hold some kind of state, and only turn them into compiler flags when all the detection is finished?

we don't know what kind of combination of features are going to be enabled through dispatch-able sources or through the build options. in the case of zsystem maybe yes, since we only deal with only three features and one kind of compiler. so instead of normalization, we will add three conditions, for example:

if cpu-baseline eq "vx" then flags = "-mzvector -march=arch11"

if cpu-baseline eq "vx vxe" then flags = "-mzvector -march=arch12"

if cpu-baseline eq "vx vxe vxe2" then flags = "-mzvector -march=arch13"

but what about the other arches/features? for more information check the following functions:

numpy/numpy/distutils/ccompiler_opt.py

Lines 1059 to 1081 in 06e37a3

def cc_normalize_flags(self, flags):
"""
Remove the conflicts that caused due gathering implied features flags.
Parameters
----------
'flags' list, compiler flags
flags should be sorted from the lowest to the highest interest.
Returns
-------
list, filtered from any conflicts.
Examples
--------
>>> self.cc_normalize_flags(['-march=armv8.2-a+fp16', '-march=armv8.2-a+dotprod'])
['armv8.2-a+fp16+dotprod']
>>> self.cc_normalize_flags(
['-msse', '-msse2', '-msse3', '-mssse3', '-msse4.1', '-msse4.2', '-mavx', '-march=core-avx2']
)
['-march=core-avx2']
"""

numpy/numpy/distutils/ccompiler_opt.py

Lines 1470 to 1475 in 06e37a3

@_Cache.me
def feature_flags(self, names):
"""
Return a list of CPU features flags sorted from the lowest
to highest interest.
"""

@pradghos
Copy link
Contributor

Thanks @seiko2plus !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
00 - Bug 36 - Build Build related PR component: numpy.distutils component: SIMD Issues in SIMD (fast instruction sets) code or machinery
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@seiko2plus @mattip @pradghos