Add missing LZ4 and ZSTD compression codec classes #25021
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4f83cd2 to
6c8b2a9
Compare
presto-hive/src/main/java/com/facebook/presto/hive/HiveCompressionCodec.java
Outdated
Show resolved
Hide resolved
hantangwangd
reviewed
May 2, 2025
| @@ -58,6 +57,7 @@ public final class IcebergSessionProperties | |||
| private static final String MINIMUM_ASSIGNED_SPLIT_WEIGHT = "minimum_assigned_split_weight"; | |||
| private static final String NESSIE_REFERENCE_NAME = "nessie_reference_name"; | |||
| private static final String NESSIE_REFERENCE_HASH = "nessie_reference_hash"; | |||
| public static final String COMPRESSION_CODEC = "compression_codec"; | |||
There was a problem hiding this comment.
Suggested change
| public static final String COMPRESSION_CODEC = "compression_codec"; | |
| static final String COMPRESSION_CODEC = "compression_codec"; |
nit: default visibility seems enough here.
These codecs are available in the writers, but don't seem to have been configured correctly. Trying to write tables with these formats previously threw errors. This change enables LZ4 and ZSTD compression for Parquet writers in Hive and Iceberg
6c8b2a9 to
97778fe
Compare
agrawalreetika
approved these changes
May 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
These codecs are available in the writers, but don't seem to have been configured correctly. Trying to write tables with these formats previously threw errors. This change enables LZ4 and ZSTD compression for Parquet writers in Iceberg and Hive
Motivation and Context
When users set the
compression_codecsession property or*.compression-codecconnector property with LZ4 or ZSTD with parquet format as the default, tables would fail to be created due to the codec being null inside HiveCompressionCodec inside of Iceberg. I couldn't find a good reason for keeping these null, so I populated the correct enum variants and added tests to ensure they worked. Since this code is shared between Iceberg and Hive connectors, I added tests for different file type and compression codec variants to ensure we have compatibility across all of the potential configuration combinations.Impact
LZ4andZSTDwhen creating iceberg tables with parquet as the default file formatTest Plan
Contributor checklist
Release Notes