Add .xz and increase performance of compression module #875
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d5bb01e to
5db3a1c
Compare
5db3a1c to
1832ef6
Compare
ddelange
commented
Aug 18, 2025
Comment on lines
+99
to
+105
| def _maybe_wrap_buffered(file_obj, mode): | ||
| # https://github.com/piskvorky/smart_open/issues/760#issuecomment-1553971657 | ||
| result = file_obj | ||
| if "b" in mode and "w" in mode: | ||
| result = io.BufferedWriter(result) | ||
| elif "b" in mode and "r" in mode: | ||
| result = io.BufferedReader(result) |
There was a problem hiding this comment.
Hi @rhpvorderman 👋
taking the liberty to tag you here for review since you originally suggested this fix in 2023 👍
There was a problem hiding this comment.
Make sure to benchmark the performance too as in CPython this buffering issue is fixed in the gzip module.
|
I'm seeing an improvement across the board for many short writes/reads 🎉 Without In [1]: import smart_open
...: from tempfile import NamedTemporaryFile as named_temporary_file
...:
...: def test_compression_extension(extension):
...: with named_temporary_file(suffix=extension) as tmp:
...: with smart_open.open(tmp.name, "w") as fout:
...: for i in range(100000):
...: fout.write("hello world\n")
...:
...: with smart_open.open(tmp.name, "r") as fin:
...: list(fin)
...:
...: with named_temporary_file(suffix=extension) as tmp:
...: with smart_open.open(tmp.name, "wb") as fout:
...: for i in range(100000):
...: fout.write(b"hello world\n")
...:
...: with smart_open.open(tmp.name, "rb") as fin:
...: list(fin)
...:
...: for extension in smart_open.compression.get_supported_extensions():
...: print(extension)
...: %timeit -n 20 test_compression_extension(extension)
...:
.bz2
329 ms ± 1.51 ms per loop (mean ± std. dev. of 7 runs, 20 loops each)
.gz
76.7 ms ± 619 μs per loop (mean ± std. dev. of 7 runs, 20 loops each)
.xz
140 ms ± 1.26 ms per loop (mean ± std. dev. of 7 runs, 20 loops each)
.zst
# UnsupportedOperation ref https://github.com/piskvorky/smart_open/pull/815With In [1]: import smart_open
...: from tempfile import NamedTemporaryFile as named_temporary_file
...:
...: def test_compression_extension(extension):
...: with named_temporary_file(suffix=extension) as tmp:
...: with smart_open.open(tmp.name, "w") as fout:
...: for i in range(100000):
...: fout.write("hello world\n")
...:
...: with smart_open.open(tmp.name, "r") as fin:
...: list(fin)
...:
...: with named_temporary_file(suffix=extension) as tmp:
...: with smart_open.open(tmp.name, "wb") as fout:
...: for i in range(100000):
...: fout.write(b"hello world\n")
...:
...: with smart_open.open(tmp.name, "rb") as fin:
...: list(fin)
...:
...: for extension in smart_open.compression.get_supported_extensions():
...: print(extension)
...: %timeit -n 20 test_compression_extension(extension)
...:
.bz2
254 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 20 loops each)
.gz
49.8 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 20 loops each)
.xz
61.9 ms ± 225 μs per loop (mean ± std. dev. of 7 runs, 20 loops each)
.zst
28.5 ms ± 239 μs per loop (mean ± std. dev. of 7 runs, 20 loops each) |
|
Released v7.3.1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes #760 and registers the
.xzcompression extension by defaultTests
Work in progress
Checklist
python update_helptext.pyin case there are API changesWorkflow