Add support for DELTA_BINARY_PACKED Parquet encoding #13391

eric-maynard · 2025-06-25T22:21:01Z

This adds support for the DELTA_BINARY_PACKED Parquet encoding.

The logic is taken from Spark's VectorizedDeltaBinaryPackedReader with adjustments made for compatibility with our existing Parquet reader.

…et-v2-refactor

…rd/iceberg into parquet-v2-delta

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

eric-maynard · 2025-07-07T17:42:07Z

@RussellSpitzer absolutely, within VectorizedDeltaEncodedValuesReader most of the divergences should be related to the different ways that Iceberg and Spark want to actually handle the decoded values. So compare the following pairs of code pointers:

Iceberg / Spark
Iceberg / Spark

Besides this, we lack support for some Spark features like skipping or Spark's readIntegersWithRebase, so lots of code is removed.

Beyond VectorizedDeltaEncodedValuesReader itself, only small changes are needed to actually plug VectorizedDeltaEncodedValuesReader into our reader stack, which already diverges from Spark's.

amogh-jahagirdar

Still going through the changes, thank you @eric-maynard

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

parquet/src/main/java/org/apache/iceberg/parquet/PageIterator.java

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

eric-maynard · 2025-07-15T14:05:33Z

Thanks for taking a look @amogh-jahagirdar, your comments should be addressed in the latest commit.

amogh-jahagirdar

Sorry for the late review @eric-maynard I checked this out locally and stepped through it, just had a minor comment but overall looks good to me. I'll hold in case @RussellSpitzer @rdblue @nastra or any others have any comments.

...java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

On further review, had a question on a code path that I want to confirm

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

RussellSpitzer

I'm on board as well, I just was hoping we could get someone who is more familiar with the spark code to take a pass as well. This looks correct to me and all the changes from the Spark version of the code look logical.

eric-maynard · 2025-07-21T23:43:34Z

Thanks @amogh-jahagirdar & @RussellSpitzer for the reviews! The PR should be updated to reflect the latest round of comments.

If this merges, I will update #13450 to reflect support for the new encoding type and take that PR out of draft.

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java

huaxingao

LGTM. Thanks @eric-maynard for the PR!

RussellSpitzer · 2025-07-30T15:07:28Z

Thanks you @eric-maynard for continuing this work and thanks @amogh-jahagirdar and @huaxingao for reviewing.

kevinjqliu · 2025-08-17T21:12:20Z

hey @eric-maynard could you backport the spark 4.0 changes to spark 3.5? We want to keep the 2 spark versions aligned in the upcoming 1.10 release. Here's some more context https://lists.apache.org/thread/8xzbg1wqft2grv8v1f13vb86vd8f7rjd

I'm happy to help with the backport too.

eric-maynard · 2025-08-18T17:41:56Z

Hey @kevinjqliu, absolutely -- please see #13859

eric-maynard added 12 commits June 10, 2025 10:26

rebase

a69ec52

lint

0bba5ef

some changes per comments

9ecc2be

Merge branch 'main' of ssh://github.com-oss/apache/iceberg into parqu…

3cd2819

…et-v2-refactor

javadoc

8d186fe

lint

5ce8913

create class

9fe0bba

remove clash

6cecf96

Merge branch 'parquet-v2-refactor' of ssh://github.com-oss/eric-mayna…

2ce2590

…rd/iceberg into parquet-v2-delta

refactoring

3aed168

clean up

98d1c5c

wire up

b72e338

github-actions bot added parquet arrow labels Jun 25, 2025

eric-maynard added 5 commits June 25, 2025 15:22

tweak header

b76cc47

check in

ec07775

resolve conflicts

c79a77c

debugging

1969466

debugging

d2b173b

github-actions bot added the spark label Jun 27, 2025

eric-maynard commented Jun 27, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Show resolved Hide resolved

github-actions bot added the data label Jul 1, 2025

eric-maynard added 6 commits July 1, 2025 13:02

debugging commit

1f219e5

move code

21c11d8

switch back to floats

e4bc23f

clean a bit

a88af2e

semistable

c375e99

polish

f8cfbb2

eric-maynard marked this pull request as ready for review July 1, 2025 20:57

stable:

9d27297

amogh-jahagirdar self-requested a review July 11, 2025 21:27

amogh-jahagirdar reviewed Jul 14, 2025

View reviewed changes

review comments

3a73ecc

eric-maynard requested a review from amogh-jahagirdar July 15, 2025 22:44

amogh-jahagirdar previously approved these changes Jul 21, 2025

View reviewed changes

...java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java Outdated Show resolved Hide resolved

amogh-jahagirdar reviewed Jul 21, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Show resolved Hide resolved

amogh-jahagirdar reviewed Jul 21, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Show resolved Hide resolved

amogh-jahagirdar self-requested a review July 21, 2025 21:23

RussellSpitzer reviewed Jul 21, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Show resolved Hide resolved

RussellSpitzer approved these changes Jul 21, 2025

View reviewed changes

eric-maynard added 3 commits July 21, 2025 16:35

amogh comments

44a81ac

russell comments

d584753

spotless

d253f1b

eric-maynard added 2 commits July 22, 2025 10:33

retry docs

e0b505b

javadoc fix

97a315e

amogh-jahagirdar approved these changes Jul 22, 2025

View reviewed changes

drexler-sky reviewed Jul 29, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Outdated Show resolved Hide resolved

huaxingao reviewed Jul 29, 2025

View reviewed changes

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Outdated Show resolved Hide resolved

...ain/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedDeltaEncodedValuesReader.java Show resolved Hide resolved

putInt

dfb7b77

huaxingao approved these changes Jul 29, 2025

View reviewed changes

RussellSpitzer merged commit c3d50e1 into apache:main Jul 30, 2025
41 of 42 checks passed

amogh-jahagirdar mentioned this pull request Aug 5, 2025

Add support for DELTA_LENGTH_BYTE_ARRAY Parquet encoding #13709

Draft

kevinjqliu mentioned this pull request Aug 19, 2025

Backport Parquet encoding tests for Spark 3.5 #13859

Merged

Add support for DELTA_BINARY_PACKED Parquet encoding #13391

Add support for DELTA_BINARY_PACKED Parquet encoding #13391

Uh oh!

Conversation

eric-maynard commented Jun 25, 2025

Uh oh!

Uh oh!

eric-maynard commented Jul 7, 2025

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eric-maynard commented Jul 15, 2025

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

eric-maynard commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RussellSpitzer commented Jul 30, 2025

Uh oh!

kevinjqliu commented Aug 17, 2025

Uh oh!

eric-maynard commented Aug 18, 2025

Uh oh!

Uh oh!

eric-maynard commented Jul 21, 2025 •

edited

Loading