Code Review: feat: support indexed v2 merge insert

Summary

This PR adds support for indexed merge insert in the v2 path, which uses scalar indices to efficiently look up target rows that match source keys instead of performing full table scans. The implementation handles both fully indexed and partially indexed datasets (where some fragments have index coverage and others don't).

P0/P1 Issues

P1: Unbounded memory usage in ReplayExec
In build_indexed_merge_physical_plan, the code uses Capacity::Unbounded for ReplayExec to avoid deadlocks with HashJoin's CollectLeft mode. While the comment explains the reasoning, this means the entire source dataset could be buffered in memory. For very large source datasets, this could cause OOM issues.

// rust/lance/src/dataset/write/merge_insert.rs
let source_replay = Arc::new(ReplayExec::new(Capacity::Unbounded, source_exec));

Consider adding documentation about this memory characteristic and/or adding a warning when source data exceeds a threshold.

P1: UpdateIf condition not properly applied
In create_indexed_action_expr, the UpdateIf case parses the condition but then ignores it:

WhenMatched::UpdateIf(condition_str) => {
    // ... parsing code ...
    if planner.create_physical_expr(&condition).is_ok() {
        // Use the matched condition combined with the parsed condition
        cases.push((matched, assign_action::Action::UpdateAll.as_literal_expr()));
    }
    // ...
}

The parsed condition should be combined with matched (e.g., matched.and(condition)) rather than being discarded. This means UpdateIf behaves the same as UpdateAll in the indexed path.

P1: Single-column key limitation may be unexpected
The indexed merge insert only supports single-column join keys, but this limitation is only enforced at runtime:

if self.params.on.len() != 1 {
    return Err(Error::invalid_input(
        "Indexed merge insert only supports single-column join keys",
        location!(),
    ));
}

Consider either:

1. Adding this check earlier in join_key_as_scalar_index so multi-column keys fall back to the non-indexed path silently, OR
2. Documenting this limitation in the public API

Minor Observations (Non-blocking)

• The new batch_size parameter is added to MergeInsertParams but not exposed in MergeInsertBuilder's public API documentation.
• Good test coverage for full, partial, and no index scenarios.
• The hybrid execution tests cover the important edge cases well.

Overall

The implementation is solid with good architectural separation between indexed and non-indexed paths. The main concerns are the memory implications of unbounded replay and the UpdateIf semantic issue.

Codecov Report

❌ Patch coverage is 76.74944% with 412 lines in your changes missing coverage. Please review.

📢 Thoughts on this report? Let us know!