Fix row to recordbatch conversion errors#36500
Open
DAlperin wants to merge 8 commits into
Open
Conversation
Three runtime errors surface when a parallel-workload run drives an Iceberg sink with a wider mix of column types: 1. "Datum Int16 does not match builder Int32Builder" — Iceberg has no smallint, so the writer context arrow schema (derived from the iceberg schema) uses Int32 while Materialize rows still carry Datum::Int16. Add a lossless Int16 -> Int32 promotion in ArrowColumn::append_datum, mirroring the existing UInt16 -> Int32 case. 2. "Field 'value' missing extension metadata" — Materialize names map fields entries/keys/values, but iceberg-rust's arrow conversion names them key_value/key/value. merge_field_metadata_recursive matched by name and silently dropped the value field's extension metadata, so ArrowBuilder later failed when constructing the inner builder. Match the map entries struct positionally instead. 3. "Failed to create EqualityDeleteWriterConfig: field_id N not found" — the planner accepted Range types as Iceberg equality delete keys, but ranges lower into Iceberg structs and iceberg-rust's RecordBatchProjector skips nested fields, so the equality field id is unreachable at runtime. Drop Range from the allow-list so the failure is caught at sink creation instead.
Explain why positional matching in `merge_map_entries_metadata` is correct by citing the Arrow `Schema.fbs` definition of `Map` as `List<entries: Struct<key, value>>` with non-enforced field names.
Pins the three runtime failures fixed in the previous commit: - `merge_map_entries_preserves_value_extension_metadata`: unit test that builds a materialize-shaped map field (entries/keys/values) and an iceberg-shaped one (key_value/key/value) and asserts the merge copies the value field's extension metadata positionally. - `test/iceberg/key-validation.td`: adds a Range key rejection block alongside the existing Map/List rejections. - `test/iceberg/catalog.td`: adds smallint and map[text=>text] sinks exercising the Int16->Int32 promotion and the entries-struct metadata merge end-to-end against a real Iceberg catalog.
DuckDB's iceberg_scan returns 0 rows for map-valued tables in the versions we test against, so the round-trip via map_keys/map_values was not actually exercising the metadata merge — the assertion just failed with no actionable signal. Check mz_sink_statuses for `running` instead: without `merge_map_entries_metadata` the sink stalls with "Field 'value' missing extension metadata" during ArrowBuilder construction, which is exactly the regression we want to pin.
builder_for_datatype was hard-coding MapFieldNames::default() (entries/keys/values) when constructing the MapBuilder, regardless of what the surrounding Schema actually said. For Iceberg the schema's map fields are key_value/key/value (preserved by merge_map_entries_metadata), so the resulting MapArray's nested DataType disagreed with the schema and RecordBatch::try_new rejected every row — the sink stalled silently and iceberg_scan saw an empty table. Read entry/key/value names off the schema's entries struct so the MapArray matches whichever convention the caller chose. COPY TO S3 PARQUET keeps building its schema with the arrow-rs defaults, so its output is unchanged. Also restores the DuckDB iceberg_scan assertion on the map_table sink in catalog.td now that the round-trip actually works.
def-
reviewed
May 11, 2026
Contributor
There was a problem hiding this comment.
Thanks for adding tests. I will rebase my iceberg sink test in parallel-workload on top of this and #36499 and report if anything else falls over. Edit: Just noticed you made it draft again, so I'll wait it's un-drafted before doing that.
Contributor
|
New failure in https://buildkite.com/materialize/nightly/builds/16384: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to some of the complicated type work.
Three runtime errors surface when a parallel-workload run drives an Iceberg sink with a wider mix of column types:
"Datum Int16 does not match builder Int32Builder" — Iceberg has no smallint, so the writer context arrow schema (derived from the iceberg schema) uses Int32 while Materialize rows still carry Datum::Int16. Add a lossless Int16 -> Int32 promotion in ArrowColumn::append_datum, mirroring the existing UInt16 -> Int32 case.
"Field 'value' missing extension metadata" — Materialize names map fields entries/keys/values, but iceberg-rust's arrow conversion names them key_value/key/value. merge_field_metadata_recursive matched by name and silently dropped the value field's extension metadata, so ArrowBuilder later failed when constructing the inner builder. Match the map entries struct positionally instead.
"Failed to create EqualityDeleteWriterConfig: field_id N not found" — the planner accepted Range types as Iceberg equality delete keys, but ranges lower into Iceberg structs and iceberg-rust's RecordBatchProjector skips nested fields, so the equality field id is unreachable at runtime. Drop Range from the allow-list so the failure is caught at sink creation instead.
Fixes SS-144, SS-143, SS-142