gh-156002: Keep reading through third-party zipfile decompressors - #157180
Merged
encukou merged 5 commits intoSep 10, 2026
Merged
Conversation
pythonGH-156003 made ZipExtFile._read1() call decompress(data, max_length) on non-deflate decompressors and consult needs_input before reading more. A decompressor installed by replacing _get_decompressor() (zipfile-zstd, zipfile-deflate64, ...) may support neither, and every read through it then failed with AttributeError. Give _decompressor_needs_input() a default for decompressors that report nothing, and only take the bounded path for decompressors that do report needs_input (the stdlib ones); others are read unbounded, as before.
|
All commit authors signed the Contributor License Agreement. |
rasmusfaber
marked this pull request as ready for review
September 8, 2026 13:59
Calling decompress() with one argument is a maintenance burden, so mark it as deprecated in 3.16.
Member
|
Thanks! I'll push to this PR directly, hope that's fine.
Could you check if this still works for you? |
Contributor
Author
Of course. I fixed a small typo that caused CI to fail, and a few other tiny nits.
Yes. This still works fine. I will see if I can get fixes merged in the third-party decompressors so they will continue to work when the fallback is removed (and be safe against CVE-2026-15310) even without a proper public API. |
|
GH-157268 is a backport of this pull request to the 3.15 branch. |
clin1234
pushed a commit
to clin1234/cpython
that referenced
this pull request
Sep 12, 2026
…ssors (pythonGH-157180) pythonGH-156003 made ZipExtFile._read1() call decompress(data, max_length) on non-deflate decompressors and consult needs_input before reading more. A decompressor installed by monkey-patching _get_decompressor() (as projects like zipfile-zstd, zipfile-deflate64, ... do) may support neither, and every read through it then failed with AttributeError. - Make LZMADecompressor.needs_input public to simplify implementation. - Make the needs_input attribute optional. - If `decompress()` fails with TypeError, try again with one argument. - Since the fallback to one-argument call is a maintenance burden, raise DeprecationWarning. - Add tests for future changes, so we can make informed decisions about breaking monkey-patchers. Co-authored-by: Petr Viktorin <[email protected]>
johnslavik
pushed a commit
to johnslavik/cpython
that referenced
this pull request
Sep 14, 2026
…ssors (pythonGH-157180) pythonGH-156003 made ZipExtFile._read1() call decompress(data, max_length) on non-deflate decompressors and consult needs_input before reading more. A decompressor installed by monkey-patching _get_decompressor() (as projects like zipfile-zstd, zipfile-deflate64, ... do) may support neither, and every read through it then failed with AttributeError. - Make LZMADecompressor.needs_input public to simplify implementation. - Make the needs_input attribute optional. - If `decompress()` fails with TypeError, try again with one argument. - Since the fallback to one-argument call is a maintenance burden, raise DeprecationWarning. - Add tests for future changes, so we can make informed decisions about breaking monkey-patchers. Co-authored-by: Petr Viktorin <[email protected]>
|
Thanks @rasmusfaber for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14. |
|
Sorry, @rasmusfaber and @encukou, I could not cleanly backport this to |
hugovk
pushed a commit
that referenced
this pull request
Sep 15, 2026
…essors (GH-157180) (#157268) Co-authored-by: rasmusfaber <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to GH-156003, as discussed in #156002 (comment).
The change to bound decompression made
ZipExtFile._read1()calldecompress(data, max_length)on non-deflate decompressors and checkneeds_input. Many third-party decompressors (zipfile-zstd, zipfile-deflate64, and others) that are installed by replacing_get_decompressor()(zipfile-zstd, zipfile-deflate64, ...) do not support those, and end up failing withAttributeError: ... has no attribute '_needs_input'.This PR:
_decompressor_needs_input()adefaultargument for decompressors that report neitherneeds_inputnor_needs_input(no three-state return);ZipExtFilecreates it (construction andseek()), whether it takes the bounded path: only decompressors that reportneeds_inputdo, which is all of the stdlib ones. Anything else is read unbounded, exactly as before gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard #156003;decompress()and noneeds_input; it fails with theAttributeErrorabove without the fix.Verified with the real packages by applying the same three hunks to the 3.13 backport (#156738): a Deflate64 archive written by 7-Zip reads through zipfile-deflate64, and a Zstandard member reads through zipfile-zstd, on Python 3.13.15 where both currently fail.
test_zipfile,test_zstd,test_zipimport,test_zipappandtest_shutilpass on a debug build of main with_zstdenabled; the*BoundedDecompressTestsstill pass for every stdlib method.Should ride along with the open backports #156737–#156741.
Claude Fable 5.1 and GPT-6 Astra both reviewed and gave input, as well as drafted PR description and commit messages.