Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Extract Iceberg type alignment logic into a pluggable interface #125

@george-zubrienko

Description

@george-zubrienko

Currently IcebergWriter performs a primitive conversion of DataRow to an Iceberg record:

  private def rowToRecord(row: DataRow, schema: Schema): GenericRecord =
    val record = GenericRecord.create(schema)
    val rowMap = row.map { cell => cell.name -> cell.value }.toMap
    record.copy(rowMap.asJava)

However, for many types emitted into DataRow, this is not as simple, starting with Optional types needing translation into nullable types. Thus, we should add a dependency of type RowConverter to CatalogWriter, and then each plugin (sql, synapse etc.), should provide its own logic for row conversions by injecting a specific implementation of a RowConverter.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions