Codestin Search App

rluvaton · 2025-09-29T16:13:31Z

Which issue does this PR close?

Closes providing the same schema that read from backward compatible parquet fails: incompatible arrow schema, expected struct got List #8495.

Rationale for this change

Fix reading old parquet files

What changes are included in this PR?

tests and the fix, but mostly tests.

Are these changes tested?

yes

Are there any user-facing changes?

No

…ated struct/primitive with provided arrow schema closes: - apache#8495

…primitive-with-inferred-schema # Conflicts: # parquet/src/arrow/schema/complex.rs

parquet/src/arrow/schema/complex.rs

…primitive-with-inferred-schema

alamb

Thank you @rluvaton -- I took a quick review of this PR and the code looks reasonable to me, but I don't understand the legacy inferring logic / problem so I can't really review this PR fully yet

Can someone help me out with a link / document that describes what the legacy inferring is? Is it https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/LogicalTypes.md?plain=1#L718-L723

Poking around that file, it looks like @etseidl may know something about this as he authored several commits, for example

apache/parquet-format#466

alamb · 2025-09-30T20:42:26Z

parquet/src/arrow/schema/complex.rs

+    /// Converts `self` into an arrow list, with its current type as the field type
+    /// accept an optional `list_data_type` to specify the type of list to create
+    ///
+    /// This is used to convert deprecated repeated columns (not in a list), into their arrow representation


Is there any reference we can add a link to (I am not familiar with the "deprecated repeated columns" you are referencing here

Maybe it is it https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/LogicalTypes.md?plain=1#L718-L723 ?

I believe it's https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/LogicalTypes.md?plain=1#L640-L643, and specifically https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/LogicalTypes.md?plain=1#L649-L650.

exactly, updated

alamb · 2025-09-30T20:48:19Z

parquet/src/arrow/schema/complex.rs

+            | Some(DataType::LargeList(field_hint))
+            | Some(DataType::FixedSizeList(field_hint, _)) => Some(field_hint.as_ref()),
+            Some(_) => unreachable!(
+                "should be validated earlier that list_data_type is only a type of list"


even though this should be impossible, I worry about panic'ing here because if there is a bug that error is more severe than "just an error"

ALso, while this may be true at the moment, I can imagine that some future refactor messes it up, in which case this may become reachable and the compiler won't complain

I would prefer returning a general_err! with some sort of "Internal error: should be validated..." type message

alamb · 2025-10-08T21:10:48Z

I will try and find time tomorrow to review this

rluvaton · 2025-10-26T10:48:07Z

@alamb Ping :)

alamb · 2025-10-28T20:27:08Z

I see this one, thank you - unfortunately I have many other PRs ahead of it in the review queue.

As I am not super familiar with this part of the spec it will likely take me longer to review as well - -maybe someone else who is more familiar can help out

martin-g · 2025-10-28T21:30:12Z

parquet/src/arrow/schema/complex.rs

                rep_level,
                def_level,
                data_type,
+                treat_repeated_as_list_arrow_hint: true,


Suggested change

treat_repeated_as_list_arrow_hint: true,

treat_repeated_as_list_arrow_hint: context.treat_repeated_as_list_arrow_hint,

this will actually lead to a bug, added a comment ad wrote a test that fail with your suggestion and pass currently

martin-g · 2025-10-28T21:35:55Z

parquet/src/arrow/schema/complex.rs

                _ => Field::new(name, data_type, nullable),
            };

            Ok(field.with_metadata(hint.metadata().clone()))


Should the extension type be added here too ? As for None below.
I.e. something like:

Suggested change

let merged = field.with_metadata(hint.metadata().clone());

try_add_extension_type(merged, parquet_type)

I think we should but not part of this PR

…primitive-with-inferred-schema

rluvaton · 2026-02-04T12:24:55Z

I see this one, thank you - unfortunately I have many other PRs ahead of it in the review queue.

As I am not super familiar with this part of the spec it will likely take me longer to review as well - -maybe someone else who is more familiar can help out

@alamb so can you please assign the relevant people, so we can merge this 4 months old PR

alamb · 2026-02-11T20:20:08Z

@alamb so can you please assign the relevant people, so we can merge this 4 months old PR

I can't assign anyone as I have no actual authority over anyone's time -- All I can do is to beg and/or cajole others into helping do so.

One thing that would help me review this pr (and perhaps others as well) is a more complete description / documentation of what this code does, with examples, as I am not all that familiar with the current schema representation of lists (and not at all familiar with the older one) would to help provide context (ideally in comments) of what this PR is doing

For example, for me to understand comments like this, I need to go into the referenced URL and then try and match the terminology used there to what is used in this repo and PR. While I can do that it will take time and time is the thing I seem to have the least of

    /// This is used to convert [deprecated repeated columns] (not in a list), into their arrow representation

Maybe it could give an example so that the code was more self contained. For example, an example of the current List representation and the old (deprecated) representation?

alamb · 2026-02-11T20:23:12Z

parquet/src/arrow/schema/complex.rs

    /// An optional [`DataType`] sourced from the embedded arrow schema
    data_type: Option<DataType>,
+
+    /// Whether to treat repeated types as list from arrow types


Should this also mention that it is for supporting "deprecated list representations" in parquet?

etseidl · 2026-02-12T00:56:44Z

I'm trying to come up to speed on this, but an early observation is that one test (backward_compat_list_struct_with_nested_repeated_primitive_respects_arrow_hint) appears to violate this line in the spec:

For all fields in the schema, implementations should use either LIST and MAP annotations or unannotated repeated fields, but not both. When using the annotations, no unannotated repeated types are allowed.

From the test in question:

        // This is a backward-compatible LIST (rule 4) where the struct element contains
        // a repeated primitive. The arrow hint specifies that the inner repeated primitive
        // should be LargeList<Int32>.
        let message_type = "
            message schema {
                optional group my_list (LIST) {
                    repeated group my_list_tuple {
                        required binary str (STRING);
                        repeated int32 values;
                    }
                }
            }
        ";

That said, plugging the tests from this PR into main without the fix yields

failures:
    arrow::schema::complex::tests::backward_compat_list_struct_with_nested_repeated_primitive_respects_arrow_hint
    arrow::schema::complex::tests::convert_schema_with_nested_repeated_struct_and_primitives
    arrow::schema::complex::tests::convert_schema_with_repeated_primitive_should_use_inferred_schema
    arrow::schema::complex::tests::convert_schema_with_repeated_primitive_should_use_inferred_schema_for_list_as_well
    arrow::schema::complex::tests::convert_schema_with_repeated_struct_and_inferred_schema
    arrow::schema::complex::tests::convert_schema_with_repeated_struct_and_inferred_schema_and_field_id

test result: FAILED. 24 passed; 6 failed; 0 ignored; 0 measured; 918 filtered out; finished in 0.64s

Other than the test mentioned above, the other failing tests seem like they should succeed.

fix(parquet): converting parquet schema with backward compatible repe…

1bcea51

…ated struct/primitive with provided arrow schema closes: - apache#8495

github-actions bot added the parquet Changes to the parquet crate label Sep 29, 2025

rluvaton added 2 commits September 29, 2025 19:25

add tests with inferring list types as well

cc5ab0d

add more tests and fix bug

fb2579a

rluvaton marked this pull request as ready for review September 29, 2025 18:33

rluvaton added 7 commits September 29, 2025 21:34

format

297cf65

added more tests as I know I have a bug

85d84b6

format

52eceed

add more tests

d299ca9

fix

1aa573f

Merge branch 'main' into fix-reading-backward-compat-repeated-struct-…

65a0bd6

…primitive-with-inferred-schema # Conflicts: # parquet/src/arrow/schema/complex.rs

align with main

ab54f02

rluvaton commented Sep 29, 2025

View reviewed changes

parquet/src/arrow/schema/complex.rs Outdated Show resolved Hide resolved

rluvaton added 7 commits September 30, 2025 15:11

set map align

f0716dd

add more tests from spark codebase for more coverage

470acbb

rename

03dfa54

format

301c7a7

remove tests from ignore and added more comments

39b9263

Merge branch 'main' into fix-reading-backward-compat-repeated-struct-…

d8d9f64

…primitive-with-inferred-schema

remove or

59303fd

alamb mentioned this pull request Sep 30, 2025

Column with List(Struct) causes failed to decode level data for struct array (regression in 56) #8404

Closed

alamb reviewed Sep 30, 2025

View reviewed changes

add link to deprecated repeated columns and replace panic with error

4d7485a

rluvaton mentioned this pull request Oct 28, 2025

Release DataFusion 51.0.0 (Nov 2025) apache/datafusion#17558

Closed

37 tasks

martin-g reviewed Oct 28, 2025

View reviewed changes

alamb mentioned this pull request Nov 9, 2025

Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) apache/datafusion#18566

Closed

41 tasks

alamb mentioned this pull request Jan 7, 2026

Release DataFusion 53.0.0 (Feb 2026 / Mar 2026) apache/datafusion#19692

Open

26 tasks

rluvaton added 3 commits February 4, 2026 11:58

Merge branch 'main' into fix-reading-backward-compat-repeated-struct-…

c1ee664

…primitive-with-inferred-schema

add comment and test why changing it to inherit the value is wrong

3eaf7a7

format

9fb3a93

alamb reviewed Feb 11, 2026

View reviewed changes

	treat_repeated_as_list_arrow_hint: true,
	treat_repeated_as_list_arrow_hint: context.treat_repeated_as_list_arrow_hint,


	let merged = field.with_metadata(hint.metadata().clone());
	try_add_extension_type(merged, parquet_type)

Conversation

rluvaton commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rluvaton Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 8, 2025

Uh oh!

rluvaton commented Oct 26, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rluvaton commented Feb 4, 2026

Uh oh!

alamb commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rluvaton commented Sep 29, 2025 •

edited

Loading

rluvaton Oct 8, 2025 •

edited

Loading