Currently IcebergWriter performs a primitive conversion of DataRow to an Iceberg record:
private def rowToRecord(row: DataRow, schema: Schema): GenericRecord =
val record = GenericRecord.create(schema)
val rowMap = row.map { cell => cell.name -> cell.value }.toMap
record.copy(rowMap.asJava)
However, for many types emitted into DataRow, this is not as simple, starting with Optional types needing translation into nullable types. Thus, we should add a dependency of type RowConverter to CatalogWriter, and then each plugin (sql, synapse etc.), should provide its own logic for row conversions by injecting a specific implementation of a RowConverter.