Is there any particular reason for using reflection here? Given it's against Gluten's own class?

Thanks @zhztheplayer you are right since this rule is Gluten-owned reflection was not needed here.

I updated this path to construct PreprocessDeltaScanWithDeletionVectors directly. To keep the direct construction compiling across the existing Spark/Delta profile matrix, I also added no-op compatibility definitions for Delta 2.3/2.4. The actual DV preprocessing behavior remains in the Delta 3.3/4.x profiles.

If reflection is inevitable, can we encapsulate the reflection code in a utility class?

@zhztheplayer I moved this reflection out of VeloxIteratorApi into DeltaSplitMetadataExtractor.

Some reflection I think is still needed here because VeloxIteratorApi is in the common Velox source set, while DeltaDeletionVectorScanInfo is only compiled in Delta-specific source profiles. The extractor now isolates that bridge and caches the class/method lookups, so the iterator path only calls the utility and does not carry the reflection block inline.

Let me know if you know a workaround?

@malinjawi does this mean we need to add new columns to table scan for Delta 3.3? if yes, can you help me understand the design here?

@malinjawi does this mean we need to add new columns to table scan for Delta 3.3? if yes, can you help me understand the design here?

@zhztheplayer

No, we do NOT add new columns to the table scan output for Delta 3.3. The key insight is we're passing DV data as file-level metadata, not as scan output columns.
The design works differently"

What Delta 3.3's PreprocessTableWithDVs does:

• Prepares Delta's internal scan metadata structures
• Makes DV information accessible via Delta's file APIs
• Does NOT inject __delta_internal_* columns into the scan output

What our code does:

• Extracts DV metadata from Delta's prepared file structures
• Passes it to Velox as per-file metadata through DeltaLocalFilesNode → Substrait protobuf
• Velox receives: serializedDeletionVector, cardinality, rowIndexFilterType for each file
• No columns added to scan output - tests verify the plan contains no __delta_internal_* columns

Delta 4.0 difference:

• Delta 4.0 prepares this metadata earlier in its own pipeline
• We just extract and forward it the same way
• The no-op rule exists because Delta 4.0 doesn't need the explicit preprocessTablesWithDVs() call

@malinjawi thanks for explanation. I may be misunderstanding but PreprocessTableWithDVs is shadowing a Delta class, we should be extremely carefully when doing so. What's the main difference for this class, compared to the one from vanilla Delta?

@zhztheplayer Good catch on the shadowing concern.

The shadowing PreprocessTableWithDVs is necessary because:

1. Mixed format handling: Delta's optimizer creates scans with DeltaParquetFileFormat, but Gluten's write path produces tables with GlutenDeltaParquetFileFormat. The DV preprocessing rule must handle BOTH formats.

2. Package-private API access: We need direct access to Delta's package-private APIs (TahoeFileIndex, deletionVectorsReadable(), DeltaSQLConf, etc.) without reflection overhead. Being in org.apache.spark.sql.delta package provides this.

We could eliminate shadowing by making GlutenDeltaParquetFileFormat extend DeltaParquetFileFormat instead of GlutenParquetFileFormat, then override only write-specific methods. This would let Delta's original PreprocessTableWithDVs handle both formats naturally.

But the tradoeff would be this breaks the clean format hierarchy separation:

• Current: GlutenParquetFileFormat → GlutenDeltaParquetFileFormat (Gluten hierarchy)
• Alternative: DeltaParquetFileFormat → GlutenDeltaParquetFileFormat (mixed hierarchy)

Right now I would say the code maintains format hierarchy separation, avoids reflection for package-private API access, and type-safe handling of both formats in DV preprocessing. But this Requires shadowing Delta's trait (maintenance burden if Delta changes internals)

Would you prefer we explore the alternative approach, or is the current shadowing acceptable given the current state?