[MRG] Include pxd-files into the installation #14896
Conversation
|
I have some questions:
|
b38ae97 to
59851bd
Compare
There was a problem hiding this comment.
Thanks for looking into this @realead !
It's a thorough approach, but I wonder if we can get away with avoiding compilation tests. In particular, adding the relevant files to Manifest.in and then add a single test that checks for the includes pxd somewere in,
sklearn/tests/test_check_build.pyi.e. that the following files exist,
$ find sklearn -iname '*.pxd'
sklearn/ensemble/_hist_gradient_boosting/common.pxd
sklearn/linear_model/sgd_fast.pxd
sklearn/neighbors/typedefs.pxd
sklearn/neighbors/dist_metrics.pxd
sklearn/neighbors/quad_tree.pxd
sklearn/svm/libsvm.pxd
sklearn/svm/liblinear.pxd
sklearn/tree/_tree.pxd
sklearn/tree/_criterion.pxd
sklearn/tree/_utils.pxd
sklearn/tree/_splitter.pxd
sklearn/utils/murmurhash.pxd
sklearn/utils/weight_vector.pxd
sklearn/utils/_cython_blas.pxd
sklearn/utils/_random.pxd
sklearn/utils/fast_dict.pxd
sklearn/utils/seq_dataset.pxdwith respect to
import sklearn
base_dir = sklearn.__file__could be enough. Since in some CI we build wheels and run tests with pytest --pyargs sklearn. So that would check that these files are included in wheels. I'm not sure there is a need to check that we can cimport them in Cython?
59851bd to
a175d2e
Compare
|
I see two problems with the approach of testing only the presence of pxd-files in the installation:
Because the behavior of a software tends to become what is tested and not what it is supposed to be, it is probably more robust to test that the pxd-files can be cimported, rather than that the pxd-files are present. But obviously it is your call, what you want to have as test. Btw, do you know, what can/should be done about failing coverage-test? |
|
I agree that dependencies between pxd is a tricky question, but these are integration/build tests that only need to be run once before a release. Making them run as part of the unit tests suite, particularly if they need to compile something each time they are run, is problematic. The resulting I would still prefer,
The next release is happening soon and I think it would be good to have it there. Also because there were some refactoring of the API, the paths to some of the pxd might have changed and it would be really useful to double-check that the included ones are still correct. Could you also please merge master in? |
test, that pxd files are in the installation and can be cimported. For that we utilize pyximport and force rebuild every time, thus the cache is not used
… special treatment (language-setting)
|
I'm not sure, why this is a problem to build and also don't think that so-files will end up in the wheels ( However, I have rebased and introduced/moved tests to a standalone tester under maint_tools. PS: not sure what this failing test is about, seems to be a missing sh-file -hopefully my PR has nothing to do with it. |
|
no longer needed. |
Reference Issues/PRs
Fixes #14847
What does this implement/fix? Explain your changes.
All pxds are now copied to the installation, so they can be reused for building Cython-extension by the user.
pyximportis used to compile the test-pyx-files during the runtime. The rebuild is forced so the cache isn't used.Any other comments?
I didn't include pxd-files from sklearn.svm (libsvm.pxd and liblinear.pxd), because in order to build them not only the h-files from
src-folder are needed, but also c and cpp files, because of this include:which leaks all definitions into the cythonized cpp-file, so the linker needs everything from cpp in order to be able to resolve all those leaked symbols.
If these pxd files should be included into the installation as well, first one needs to introduce libsvm_helper.h and build libsvm_helper.c as a source file and be including it into the pxd (and the same for liblinear.pxd)