Thanks to visit codestin.com
Credit goes to github.com

Skip to content

FIX python-lz4 version comparison - #847

Merged
ogrisel merged 4 commits into
joblib:masterfrom
sveitser:python-lz4-version-fix
May 20, 2019
Merged

FIX python-lz4 version comparison#847
ogrisel merged 4 commits into
joblib:masterfrom
sveitser:python-lz4-version-fix

Conversation

@sveitser

@sveitser sveitser commented Feb 16, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

The python-lz4 package has its version string prefixed with "v". As a result on nixos a test fails.

This commit changes to pkg_resources.parse_version which normalizes
the two schemas correctly and seems to be the recommended way to
compare versions in python at the moment.

Nixos build log https://hydra.nixos.org/build/88820470/nixlog/2.

Failing test:


=================================== FAILURES ===================================
_____________________________ test_lz4_compression _____________________________

tmpdir = local('/build/pytest-of-nixbld/pytest-0/test_lz4_compression0')

    @with_lz4
    def test_lz4_compression(tmpdir):
        # Check that lz4 can be used when dependency is available.
        import lz4.frame
        compressor = 'lz4'
        assert compressor in _COMPRESSORS
        assert _COMPRESSORS[compressor].fileobj_factory == lz4.frame.LZ4FrameFile
    
        fname = tmpdir.join('test.pkl').strpath
        data = 'test data'
>       numpy_pickle.dump(data, fname, compress=compressor)

joblib/test/test_numpy_pickle.py:1047: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
joblib/numpy_pickle.py:498: in dump
    compress_level)) as f:
joblib/numpy_pickle_utils.py:192: in _write_fileobject
    filename, compresslevel=compresslevel)
joblib/compressor.py:234: in compressor_file
    self._check_versions()
joblib/compressor.py:229: in _check_versions
    if lz4 is None or LooseVersion(lz4.__version__) < LooseVersion('0.19'):
/nix/store/6la3fblc497pacyy0vxhnjq1l1wlb5w7-python3-3.7.2/lib/python3.7/distutils/version.py:52: in __lt__
    c = self._cmp(other)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LooseVersion ('v2.1.2'), other = LooseVersion ('0.19')

    def _cmp (self, other):
        if isinstance(other, str):
            other = LooseVersion(other)
    
        if self.version == other.version:
            return 0
>       if self.version < other.version:
E       TypeError: '<' not supported between instances of 'str' and 'int'

@sveitser
FIX python-lz4 version comparison
d3235fd 
The python-lz4 package has its version string prefixed with "v".
As a result on nixos a test fails.

This commit changes to pkg_resources.parse_version which normalizes
the two schemas correctly and seems to be the recommended way to
compare versions in python at the moment.

Nixos build log https://hydra.nixos.org/build/88820470/nixlog/2.
@sveitser
FIX numpy version comparison
884c92c
@codecov

codecov Bot commented Feb 16, 2019

Copy link
Copy Markdown

Codecov Report

Merging #847 into master will decrease coverage by 1.91%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #847      +/-   ##
==========================================
- Coverage   94.55%   92.63%   -1.92%     
==========================================
  Files          45       45              
  Lines        6407     6407              
==========================================
- Hits         6058     5935     -123     
- Misses        349      472     +123
Impacted Files Coverage Δ
joblib/compressor.py 90.93% <100%> (-2.59%) ⬇️
joblib/backports.py 93.75% <100%> (+54.16%) ⬆️
joblib/numpy_pickle_compat.py 49% <0%> (-42%) ⬇️
joblib/test/test_numpy_pickle.py 88.5% <0%> (-9.86%) ⬇️
joblib/testing.py 87.5% <0%> (-7.5%) ⬇️
joblib/_dask.py 91.52% <0%> (-3.96%) ⬇️
joblib/numpy_pickle.py 94.58% <0%> (-3.95%) ⬇️
joblib/_multiprocessing_helpers.py 76.92% <0%> (-3.85%) ⬇️
joblib/numpy_pickle_utils.py 90.32% <0%> (-3.23%) ⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a99273b...884c92c. Read the comment docs.

@codecov

codecov Bot commented Feb 16, 2019 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #847 into master will increase coverage by 11.2%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #847      +/-   ##
==========================================
+ Coverage   84.15%   95.36%   +11.2%     
==========================================
  Files          45       45              
  Lines        6406     6407       +1     
==========================================
+ Hits         5391     6110     +719     
+ Misses       1015      297     -718
Impacted Files Coverage Δ
joblib/compressor.py 93.52% <100%> (+11%) ⬆️
joblib/backports.py 93.75% <100%> (+58.33%) ⬆️
joblib/test/test_memmapping.py 99.41% <0%> (+0.29%) ⬆️
joblib/_parallel_backends.py 93.6% <0%> (+0.79%) ⬆️
joblib/memory.py 95.69% <0%> (+1.34%) ⬆️
joblib/test/test_hashing.py 98.88% <0%> (+1.67%) ⬆️
joblib/logger.py 86.84% <0%> (+2.63%) ⬆️
joblib/parallel.py 97.28% <0%> (+2.71%) ⬆️
joblib/_memmapping_reducer.py 95% <0%> (+2.77%) ⬆️
joblib/my_exceptions.py 98.11% <0%> (+3.77%) ⬆️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3169ca6...6dc4a7c. Read the comment docs.

@FlorianFranzen

FlorianFranzen commented Mar 9, 2019

Copy link
Copy Markdown

Would be nice if we can get this merged.

@aabadie

aabadie commented Mar 18, 2019

Copy link
Copy Markdown
Contributor

LGTM

@ogrisel any idea why the scikit-learn tests are failing ? I tried with 0.20.3 on my fork but have another failing test, see https://travis-ci.org/aabadie/joblib/jobs/507830463 and aabadie@17b3ee3
This is also broken on master btw.

@ogrisel

ogrisel commented May 20, 2019

Copy link
Copy Markdown
Contributor

I merged @lesteve's #861 to master and merged master into this branch to check whether that fixes the problem.

@ogrisel

ogrisel commented May 20, 2019

Copy link
Copy Markdown
Contributor

The windows error is unrelated, let's merge.

@ogrisel
ogrisel merged commit efe896a into joblib:master May 20, 2019
pierreglaser pushed a commit to pierreglaser/joblib that referenced this pull request Jun 12, 2019
@sveitser @pierreglaser
FIX python-lz4 version comparison (joblib#847)
0321509
@cgohlke cgohlke mentioned this pull request Dec 4, 2019
Closed
ogrisel added a commit to ogrisel/joblib that referenced this pull request Dec 10, 2019
@ogrisel
Revert "FIX python-lz4 version comparison (joblib#847)"
a4c5aa4 
This reverts commit efe896a.
ogrisel added a commit that referenced this pull request Dec 10, 2019
@ogrisel
Revert pkg resources (#971)
064b102 
* Revert "FIX python-lz4 version comparison (#847)"

This reverts commit efe896a.

* Manually deal with LZ4 version issues

* Changelog update

* Ignore vscode configuration
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Dec 14, 2019
adam
py-joblib: updated to 0.14.1
15195f5 
Release 0.14.1

Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case:

allow for a suitable number of threads for numba (NUMBA_NUM_THREADS);
enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1)
joblib/joblib#951

Fix a regression where the loky backend was not reusing previously spawned workers. joblib/joblib#968

Revert joblib/joblib#847 to avoid using pkg_resources that introduced a performance regression under Windows: joblib/joblib#965
msk pushed a commit to msk/pkgsrc that referenced this pull request May 11, 2026
adam
py-joblib: updated to 0.14.1
1daca63 
Release 0.14.1

Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case:

allow for a suitable number of threads for numba (NUMBA_NUM_THREADS);
enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1)
joblib/joblib#951

Fix a regression where the loky backend was not reusing previously spawned workers. joblib/joblib#968

Revert joblib/joblib#847 to avoid using pkg_resources that introduced a performance regression under Windows: joblib/joblib#965
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request May 13, 2026
adam
py-joblib: updated to 0.14.1
8c40550 
Release 0.14.1

Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case:

allow for a suitable number of threads for numba (NUMBA_NUM_THREADS);
enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1)
joblib/joblib#951

Fix a regression where the loky backend was not reusing previously spawned workers. joblib/joblib#968

Revert joblib/joblib#847 to avoid using pkg_resources that introduced a performance regression under Windows: joblib/joblib#965
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sveitser @FlorianFranzen @aabadie @ogrisel