Add file compression option to run() and write_tables() #21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Without compression, write_files() produces massive output files on the order of gigabytes per run. Adding a simple HDF5-standard zlib compression parameter cuts output file size by 75%, and adds a few seconds to I/O while writing. zlib is not a very fast compression algorithm, but it is included as standard in every HDF5 implementation.
|
This is great, thanks! I tried this out and see the ~80% reduction in the size of output .h5. In my case, the write-out with compression took about 6 seconds longer. |
|
That's awesome. Does the compression impact read times at all? |
|
@bridwell Does not seem to affect read time on my machine -- at least, it's within margin of error given everything else happening on my desktop. (edit) -- in fact, sometimes it's faster since there is so much less file to read. |
|
Thanks, Billy. I've used the compression library on large HDF5 files before, but had not checked to see that we were not using it as a default in orca. Good call. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Without compression,
orca.run()andorca.write_files()produce massive output files, several gigabytes in size.Adding a simple HDF5-standard zlib compression parameter cuts output file size by 80%, while adding just a few seconds to I/O while writing.
The new
compressparameter is optional and the default is False, so existing UDST code does not need to be modified and will not be affected by this new option, if people do not want smaller files.zlib is not a very fast compression algorithm, but it is included as standard in every HDF5 implementation. It's the correct choice.